By type
*** retrieve Truststore and Keystore password
*** actual launch of AAI Resources
2022-03-22 03:09:31.866 INFO --- [ main] org.onap.aai.util.AAIConfig : Initializing AAIConfig
2022-03-22 03:09:31.921 DEBUG --- [ main] org.onap.aai.util.AAIConfig : Reloading config from /opt/app/aai-traversal/./resources/etc/appprops/aaiconfig.properties
2022-03-22 03:09:31.922 INFO --- [ main] org.onap.aai.util.AAIConfig : Found the aaiconfig.properties in the following location: /opt/app/aai-traversal/./resources/etc/appprops/aaiconfig.properties
2022-03-22 03:09:31.936 INFO --- [ main] org.onap.aai.util.AAIConfig : A&AI Server Node Name = aai.config.nodename
2022-03-22 03:09:38.227 WARN 10 --- [ main] o.s.b.l.logback.LogbackLoggingSystem : Ignoring 'logback.configurationFile' system property. Please use 'logging.config' instead.
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.12.RELEASE)
03:10:00,484 |-INFO in LogbackRequestLog - Will use configuration resource [/localhost-access-logback.xml]
03:10:00,498 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - debug attribute not set
03:10:00,498 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
03:10:00,498 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ACCESS]
03:10:00,504 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@1733581655 - No compression will be used
03:10:00,504 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@1733581655 - Will use the pattern /opt/app/aai-traversal/logs/ajsc-jetty/localhost_access.log.%d{yyyy-MM-dd} for the active file
03:10:00,505 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - The date pattern is 'yyyy-MM-dd' from file name pattern '/opt/app/aai-traversal/logs/ajsc-jetty/localhost_access.log.%d{yyyy-MM-dd}'.
03:10:00,505 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Roll-over at midnight.
03:10:00,506 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Setting initial period to Tue Mar 22 03:07:46 GMT 2022
03:10:00,539 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - Active log file name: /opt/app/aai-traversal/logs/ajsc-jetty/localhost_access.log
03:10:00,539 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - File property is set to [/opt/app/aai-traversal/logs/ajsc-jetty/localhost_access.log]
03:10:00,539 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ACCESS] to null
03:10:00,539 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - End of configuration.
03:10:00,539 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@6c2c1385 - Registering current configuration as safe fallback point
2022-03-22 03:10:01.088 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED CLASSES [class org.onap.aai.rest.search.ModelAndNamedQueryRestProvider, class org.onap.aai.rest.util.EchoResponse, class org.onap.aai.rest.CQ2GremlinTest, class org.onap.aai.rest.QueryConsumer, class org.onap.aai.rest.RecentAPIConsumer, class org.onap.aai.rest.search.SearchProvider, class org.onap.aai.rest.CQ2Gremlin, class org.onap.aai.rest.DslConsumer]
2022-03-22 03:10:01.097 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.OneWaySslAuthorization result=true
2022-03-22 03:10:01.099 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.VersionInterceptor result=true
2022-03-22 03:10:01.099 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.ResponseHeaderManipulation result=true
2022-03-22 03:10:01.100 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.ResponseTransactionLogging result=true
2022-03-22 03:10:01.101 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestModification result=true
2022-03-22 03:10:01.101 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestHeaderManipulation result=true
2022-03-22 03:10:01.102 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.HttpHeaderInterceptor result=true
2022-03-22 03:10:01.103 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.InvalidResponseStatus result=true
2022-03-22 03:10:01.103 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestTransactionLogging result=true
2022-03-22 03:10:01.104 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.HeaderValidation result=true
2022-03-22 03:10:01.108 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.VersionLatestInterceptor result=true
2022-03-22 03:10:01.108 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RetiredInterceptor result=true
2022-03-22 03:10:01.118 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.OneWaySslAuthorization result=true
2022-03-22 03:10:01.119 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.VersionInterceptor result=true
2022-03-22 03:10:01.120 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.ResponseHeaderManipulation result=true
2022-03-22 03:10:01.121 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.ResponseTransactionLogging result=true
2022-03-22 03:10:01.121 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestModification result=true
2022-03-22 03:10:01.122 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestHeaderManipulation result=true
2022-03-22 03:10:01.123 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.HttpHeaderInterceptor result=true
2022-03-22 03:10:01.124 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.InvalidResponseStatus result=true
2022-03-22 03:10:01.124 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestTransactionLogging result=true
2022-03-22 03:10:01.125 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.HeaderValidation result=true
2022-03-22 03:10:01.126 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.VersionLatestInterceptor result=true
2022-03-22 03:10:01.126 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RetiredInterceptor result=true
2022-03-22 03:10:01.128 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.post.InvalidResponseStatus
2022-03-22 03:10:01.128 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.RequestTransactionLogging
2022-03-22 03:10:01.129 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.HeaderValidation
2022-03-22 03:10:01.130 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.post.ResponseTransactionLogging
2022-03-22 03:10:01.130 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.post.ResponseHeaderManipulation
2022-03-22 03:10:01.131 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.HttpHeaderInterceptor
2022-03-22 03:10:01.132 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.VersionLatestInterceptor
2022-03-22 03:10:01.132 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.OneWaySslAuthorization
2022-03-22 03:10:01.133 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.RetiredInterceptor
2022-03-22 03:10:01.134 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.VersionInterceptor
2022-03-22 03:10:01.134 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.RequestHeaderManipulation
2022-03-22 03:10:01.135 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.RequestModification
2022-03-22 03:10:02.113 DEBUG 10 --- [ main] i.m.c.u.i.logging.InternalLoggerFactory : Using SLF4J as the default logging framework
2022-03-22 03:10:09.299 INFO 10 --- [ main] org.onap.aai.config.NodesConfiguration : Translator is SchemaServiceTranslator
2022-03-22 03:10:11.677 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:11.677 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:11.678 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:11.681 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:10:11.686 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/versions, BODY=
2022-03-22 03:10:12.283 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:12.283 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:12.284 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:12.288 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:14.089 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:14.314 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Processing the translator
2022-03-22 03:10:14.317 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv11
2022-03-22 03:10:14.318 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:14.511 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:14.512 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:14.512 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:14.512 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:14.587 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:14.740 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:14.741 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:14.741 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:24.613 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 222
2022-03-22 03:10:24.691 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv12
2022-03-22 03:10:24.700 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:24.706 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:24.706 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:24.706 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:24.706 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:24.773 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:24.929 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:24.930 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:24.930 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:28.405 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 235
2022-03-22 03:10:28.431 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv13
2022-03-22 03:10:28.432 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:28.435 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:28.435 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:28.436 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:28.436 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:28.479 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:28.537 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:28.539 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:28.540 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:31.526 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 237
2022-03-22 03:10:31.601 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv14
2022-03-22 03:10:31.603 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:31.606 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:31.607 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:31.607 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:31.607 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:31.663 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:31.733 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:31.735 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:31.735 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:35.236 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 272
2022-03-22 03:10:35.283 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv15
2022-03-22 03:10:35.284 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:35.289 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:35.289 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:35.289 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:35.289 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:35.350 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:35.484 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:35.488 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:35.497 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:38.991 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 279
2022-03-22 03:10:39.007 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv16
2022-03-22 03:10:39.008 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:39.011 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:39.012 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:39.012 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:39.012 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:39.100 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:39.131 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:39.132 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:39.133 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:42.458 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 281
2022-03-22 03:10:42.478 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv17
2022-03-22 03:10:42.479 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:42.482 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:42.482 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:42.482 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:42.482 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:42.551 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:42.711 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:42.712 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:42.712 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:46.251 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 281
2022-03-22 03:10:46.269 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv18
2022-03-22 03:10:46.280 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:46.286 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:46.287 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:46.287 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:46.287 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:46.337 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:46.437 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:46.439 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:46.444 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:50.591 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 281
2022-03-22 03:10:50.615 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv19
2022-03-22 03:10:50.617 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:50.627 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:50.627 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:50.642 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:50.643 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:50.750 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:50.831 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:50.832 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:50.832 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:54.744 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 301
2022-03-22 03:10:54.762 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv20
2022-03-22 03:10:54.763 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:54.767 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:54.767 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:54.767 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:54.767 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:54.836 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:54.905 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:54.907 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:54.908 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:10:57.750 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 305
2022-03-22 03:10:57.797 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv21
2022-03-22 03:10:57.799 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:10:57.803 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:10:57.803 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:10:57.804 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:10:57.804 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:10:57.859 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:10:57.937 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:10:57.938 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:10:57.938 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:11:01.221 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 371
2022-03-22 03:11:01.241 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv22
2022-03-22 03:11:01.242 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:11:01.245 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:01.245 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:01.246 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:01.247 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:01.347 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:01.435 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:01.436 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:11:01.437 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:11:05.223 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 378
2022-03-22 03:11:05.246 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv23
2022-03-22 03:11:05.247 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:11:05.250 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:05.251 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:05.251 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:05.251 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:05.313 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:05.423 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:05.424 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:11:05.425 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:11:08.668 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 378
2022-03-22 03:11:08.691 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv24
2022-03-22 03:11:08.692 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2022-03-22 03:11:08.695 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:08.695 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:08.695 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:08.695 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:08.760 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:08.838 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:08.891 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2022-03-22 03:11:08.891 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2022-03-22 03:11:12.662 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 406
2022-03-22 03:11:12.703 DEBUG 10 --- [ main] org.onap.aai.TraversalApp : AAI Server initialization started...
2022-03-22 03:11:12.704 DEBUG 10 --- [ main] org.onap.aai.TraversalApp : Starting AAIGraph connections and the NodeInjestor
2022-03-22 03:11:12.704 INFO 10 --- [ main] org.onap.aai.util.AAIConfig : Initializing AAIConfig
2022-03-22 03:11:12.704 DEBUG 10 --- [ main] org.onap.aai.util.AAIConfig : Reloading config from /opt/app/aai-traversal/./resources/etc/appprops/aaiconfig.properties
2022-03-22 03:11:12.705 INFO 10 --- [ main] org.onap.aai.util.AAIConfig : Found the aaiconfig.properties in the following location: /opt/app/aai-traversal/./resources/etc/appprops/aaiconfig.properties
2022-03-22 03:11:12.706 INFO 10 --- [ main] org.onap.aai.util.AAIConfig : A&AI Server Node Name = aai.config.nodename
2022-03-22 03:11:13.450 INFO 10 --- [ main] com.datastax.driver.core : DataStax Java driver 3.6.0 for Apache Cassandra
2022-03-22 03:11:13.465 INFO 10 --- [ main] c.d.driver.core.GuavaCompatibility : Detected Guava >= 19 in the classpath, using modern compatibility layer
2022-03-22 03:11:13.467 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.NEW_NODE_DELAY_SECONDS is undefined, using default value 1
2022-03-22 03:11:13.467 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.NOTIF_LOCK_TIMEOUT_SECONDS is undefined, using default value 60
2022-03-22 03:11:13.504 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.USE_NATIVE_CLOCK is undefined, using default value true
2022-03-22 03:11:14.079 INFO 10 --- [ main] com.datastax.driver.core.ClockFactory : Using native clock to generate timestamps.
2022-03-22 03:11:14.101 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.NON_BLOCKING_EXECUTOR_SIZE is undefined, using default value 2
2022-03-22 03:11:14.260 DEBUG 10 --- [ main] com.datastax.driver.core.Cluster : Starting new cluster with contact points [cassandra/10.233.68.68:9042]
2022-03-22 03:11:14.343 DEBUG 10 --- [ main] i.n.u.i.logging.InternalLoggerFactory : Using SLF4J as the default logging framework
2022-03-22 03:11:14.402 DEBUG 10 --- [ main] i.n.u.internal.InternalThreadLocalMap : -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2022-03-22 03:11:14.402 DEBUG 10 --- [ main] i.n.u.internal.InternalThreadLocalMap : -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2022-03-22 03:11:14.484 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : -Dio.netty.noUnsafe: false
2022-03-22 03:11:14.484 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : Java version: 8
2022-03-22 03:11:14.490 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : sun.misc.Unsafe.theUnsafe: available
2022-03-22 03:11:14.492 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : sun.misc.Unsafe.copyMemory: available
2022-03-22 03:11:14.495 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : java.nio.Buffer.address: available
2022-03-22 03:11:14.497 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : direct buffer constructor: available
2022-03-22 03:11:14.500 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : java.nio.Bits.unaligned: available, true
2022-03-22 03:11:14.501 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
2022-03-22 03:11:14.501 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : java.nio.DirectByteBuffer.(long, int): available
2022-03-22 03:11:14.501 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : sun.misc.Unsafe: available
2022-03-22 03:11:14.502 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2022-03-22 03:11:14.502 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.bitMode: 64 (sun.arch.data.model)
2022-03-22 03:11:14.507 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.maxDirectMemory: 1020067840 bytes
2022-03-22 03:11:14.508 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.uninitializedArrayAllocationThreshold: -1
2022-03-22 03:11:14.515 DEBUG 10 --- [ main] io.netty.util.internal.CleanerJava6 : java.nio.ByteBuffer.cleaner(): available
2022-03-22 03:11:14.516 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.noPreferDirect: false
2022-03-22 03:11:14.529 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.FORCE_NIO is undefined, using default value false
2022-03-22 03:11:14.550 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
2022-03-22 03:11:14.550 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : -Dio.netty.native.deleteLibAfterLoading: true
2022-03-22 03:11:14.550 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : -Dio.netty.native.tryPatchShadedId: true
2022-03-22 03:11:14.560 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : Unable to load the library 'netty_transport_native_epoll_x86_64', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:290)
at io.netty.channel.epoll.Native.(Native.java:70)
at io.netty.channel.epoll.Epoll.(Epoll.java:39)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
2022-03-22 03:11:14.563 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : netty_transport_native_epoll_x86_64 cannot be loaded from java.library.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:290)
at io.netty.channel.epoll.Native.(Native.java:70)
at io.netty.channel.epoll.Epoll.(Epoll.java:39)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 64 common frames omitted
2022-03-22 03:11:14.631 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : Unable to load the library '/tmp/libnetty_transport_native_epoll_x86_644668937344034760256.so', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: unsupported JNI version 0xFFFFFFFF required by /tmp/libnetty_transport_native_epoll_x86_644668937344034760256.so
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:203)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:290)
at io.netty.channel.epoll.Native.(Native.java:70)
at io.netty.channel.epoll.Epoll.(Epoll.java:39)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
2022-03-22 03:11:14.639 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : Unable to load the library 'netty_transport_native_epoll', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:293)
at io.netty.channel.epoll.Native.(Native.java:70)
at io.netty.channel.epoll.Epoll.(Epoll.java:39)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
2022-03-22 03:11:14.640 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : netty_transport_native_epoll cannot be loaded from java.library.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:293)
at io.netty.channel.epoll.Native.(Native.java:70)
at io.netty.channel.epoll.Epoll.(Epoll.java:39)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 64 common frames omitted
2022-03-22 03:11:14.644 WARN 10 --- [ main] com.datastax.driver.core.NettyUtil : Found Netty's native epoll transport in the classpath, but epoll is not available. Using NIO instead.
java.lang.UnsatisfiedLinkError: unsupported JNI version 0xFFFFFFFF required by /tmp/libnetty_transport_native_epoll_x86_644668937344034760256.so
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:203)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:290)
at io.netty.channel.epoll.Native.(Native.java:70)
at io.netty.channel.epoll.Epoll.(Epoll.java:39)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
Suppressed: java.lang.UnsatisfiedLinkError: unsupported JNI version 0xFFFFFFFF required by /tmp/libnetty_transport_native_epoll_x86_644668937344034760256.so
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 64 common frames omitted
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
... 63 common frames omitted
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 64 common frames omitted
Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_transport_native_epoll
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:224)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:293)
... 62 common frames omitted
Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_transport_native_epoll.so
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
... 63 common frames omitted
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
... 63 common frames omitted
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 64 common frames omitted
2022-03-22 03:11:14.670 DEBUG 10 --- [ main] i.n.channel.MultithreadEventLoopGroup : -Dio.netty.eventLoopThreads: 4
2022-03-22 03:11:14.714 DEBUG 10 --- [ main] io.netty.channel.nio.NioEventLoop : -Dio.netty.noKeySetOptimization: false
2022-03-22 03:11:14.714 DEBUG 10 --- [ main] io.netty.channel.nio.NioEventLoop : -Dio.netty.selectorAutoRebuildThreshold: 512
2022-03-22 03:11:14.754 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : org.jctools-core.MpscChunkedArrayQueue: available
2022-03-22 03:11:14.823 DEBUG 10 --- [ main] io.netty.util.ResourceLeakDetector : -Dio.netty.leakDetection.level: simple
2022-03-22 03:11:14.824 DEBUG 10 --- [ main] io.netty.util.ResourceLeakDetector : -Dio.netty.leakDetection.targetRecords: 4
2022-03-22 03:11:14.833 DEBUG 10 --- [ main] i.n.util.ResourceLeakDetectorFactory : Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@5df17e60
2022-03-22 03:11:14.849 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.EXTENDED_PEER_CHECK is undefined, using default value true
2022-03-22 03:11:14.972 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:14.981 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.DISABLE_COALESCING is undefined, using default value false
2022-03-22 03:11:14.982 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.FLUSHER_SCHEDULE_PERIOD_NS is undefined, using default value 10000
2022-03-22 03:11:14.982 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.FLUSHER_RUN_WITHOUT_WORK_TIMES is undefined, using default value 5
2022-03-22 03:11:15.148 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.numHeapArenas: 4
2022-03-22 03:11:15.149 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.numDirectArenas: 4
2022-03-22 03:11:15.149 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.pageSize: 8192
2022-03-22 03:11:15.149 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.maxOrder: 11
2022-03-22 03:11:15.150 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.chunkSize: 16777216
2022-03-22 03:11:15.150 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.tinyCacheSize: 512
2022-03-22 03:11:15.150 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.smallCacheSize: 256
2022-03-22 03:11:15.150 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.normalCacheSize: 64
2022-03-22 03:11:15.150 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2022-03-22 03:11:15.150 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.cacheTrimInterval: 8192
2022-03-22 03:11:15.150 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.cacheTrimIntervalMillis: 0
2022-03-22 03:11:15.150 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.useCacheForAllThreads: true
2022-03-22 03:11:15.150 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
2022-03-22 03:11:15.249 DEBUG 10 --- [ main] io.netty.channel.DefaultChannelId : -Dio.netty.processId: 10 (auto-detected)
2022-03-22 03:11:15.258 DEBUG 10 --- [ main] io.netty.util.NetUtil : -Djava.net.preferIPv4Stack: false
2022-03-22 03:11:15.258 DEBUG 10 --- [ main] io.netty.util.NetUtil : -Djava.net.preferIPv6Addresses: false
2022-03-22 03:11:15.266 DEBUG 10 --- [ main] io.netty.util.NetUtil : Loopback interface: lo (lo, 127.0.0.1)
2022-03-22 03:11:15.269 DEBUG 10 --- [ main] io.netty.util.NetUtil : /proc/sys/net/core/somaxconn: 4096
2022-03-22 03:11:15.273 DEBUG 10 --- [ main] io.netty.channel.DefaultChannelId : -Dio.netty.machineId: 02:5b:7b:ff:fe:e2:dd:d8 (auto-detected)
2022-03-22 03:11:15.375 DEBUG 10 --- [ main] io.netty.buffer.ByteBufUtil : -Dio.netty.allocator.type: pooled
2022-03-22 03:11:15.376 DEBUG 10 --- [ main] io.netty.buffer.ByteBufUtil : -Dio.netty.threadLocalDirectBufferSize: 0
2022-03-22 03:11:15.376 DEBUG 10 --- [ main] io.netty.buffer.ByteBufUtil : -Dio.netty.maxThreadLocalCharBufferSize: 16384
2022-03-22 03:11:15.485 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:11:15.697 DEBUG 10 --- [er-nio-worker-0] io.netty.util.Recycler : -Dio.netty.recycler.maxCapacityPerThread: 4096
2022-03-22 03:11:15.697 DEBUG 10 --- [er-nio-worker-0] io.netty.util.Recycler : -Dio.netty.recycler.maxSharedCapacityFactor: 2
2022-03-22 03:11:15.697 DEBUG 10 --- [er-nio-worker-0] io.netty.util.Recycler : -Dio.netty.recycler.linkCapacity: 16
2022-03-22 03:11:15.697 DEBUG 10 --- [er-nio-worker-0] io.netty.util.Recycler : -Dio.netty.recycler.ratio: 8
2022-03-22 03:11:15.739 DEBUG 10 --- [er-nio-worker-0] io.netty.buffer.AbstractByteBuf : -Dio.netty.buffer.checkAccessible: true
2022-03-22 03:11:15.739 DEBUG 10 --- [er-nio-worker-0] io.netty.buffer.AbstractByteBuf : -Dio.netty.buffer.checkBounds: true
2022-03-22 03:11:15.739 DEBUG 10 --- [er-nio-worker-0] i.n.util.ResourceLeakDetectorFactory : Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@43e2a232
2022-03-22 03:11:15.874 DEBUG 10 --- [er-nio-worker-0] c.datastax.driver.core.SystemProperties : com.datastax.driver.NATIVE_TRANSPORT_MAX_FRAME_SIZE_IN_MB is undefined, using default value 256
2022-03-22 03:11:16.070 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2022-03-22 03:11:16.073 DEBUG 10 --- [ main] c.d.driver.core.ControlConnection : [Control connection] Refreshing node list and token map
2022-03-22 03:11:16.262 DEBUG 10 --- [ main] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema
2022-03-22 03:11:16.530 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [Control connection] established to cassandra/10.233.68.68:9042
2022-03-22 03:11:16.532 INFO 10 --- [ main] c.d.d.c.p.DCAwareRoundRobinPolicy : Using data-center name 'Pod' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2022-03-22 03:11:16.536 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host /10.233.66.75:9042 added
2022-03-22 03:11:16.536 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host /10.233.71.64:9042 added
2022-03-22 03:11:16.537 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host cassandra/10.233.68.68:9042 added
2022-03-22 03:11:16.548 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.CHECK_IO_DEADLOCKS is undefined, using default value true
2022-03-22 03:11:16.556 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:16.563 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:11:16.573 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:16.583 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] preparing to open 1 new connections, total = 2
2022-03-22 03:11:16.588 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:11:16.598 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-1, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:11:16.600 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-1, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:11:16.605 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preventing new connections for the next 1000 ms
2022-03-22 03:11:16.606 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-1, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:11:16.621 DEBUG 10 --- [luster-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host DOWN
2022-03-22 03:11:16.622 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-1, inFlight=0, closed=true] closing connection
2022-03-22 03:11:16.628 DEBUG 10 --- [luster-worker-0] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 03:11:16.630 WARN 10 --- [er-nio-worker-0] c.d.driver.core.HostConnectionPool : Error creating connection to /10.233.71.64:9042
com.datastax.driver.core.exceptions.TransportException: [/10.233.71.64:9042] Cannot connect
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:224)
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:194)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.71.64:9042
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 03:11:16.636 WARN 10 --- [er-nio-worker-0] com.datastax.driver.core.Session : Error creating pool to /10.233.71.64:9042
com.datastax.driver.core.exceptions.ConnectionException: [/10.233.71.64:9042] Pool was closed during initialization
at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173)
at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1773)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.google.common.util.concurrent.CollectionFuture$CollectionFutureRunningState.handleAllCompleted(CollectionFuture.java:76)
at com.google.common.util.concurrent.AggregateFuture$RunningState.processCompleted(AggregateFuture.java:255)
at com.google.common.util.concurrent.AggregateFuture$RunningState.decrementCountAndMaybeComplete(AggregateFuture.java:242)
at com.google.common.util.concurrent.AggregateFuture$RunningState.access$300(AggregateFuture.java:91)
at com.google.common.util.concurrent.AggregateFuture$RunningState$1.run(AggregateFuture.java:146)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.completeWithFuture(AbstractFuture.java:730)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:666)
at com.google.common.util.concurrent.Futures$AsyncCatchingFuture.doFallback(Futures.java:826)
at com.google.common.util.concurrent.Futures$AsyncCatchingFuture.doFallback(Futures.java:813)
at com.google.common.util.concurrent.Futures$AbstractCatchingFuture.run(Futures.java:789)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.completeWithFuture(AbstractFuture.java:730)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:666)
at com.google.common.util.concurrent.Futures$AsyncCatchingFuture.doFallback(Futures.java:826)
at com.google.common.util.concurrent.Futures$AsyncCatchingFuture.doFallback(Futures.java:813)
at com.google.common.util.concurrent.Futures$AbstractCatchingFuture.run(Futures.java:789)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1405)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:53)
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:222)
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:194)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 03:11:16.677 DEBUG 10 --- [luster-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] starting reconnection attempt
2022-03-22 03:11:17.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] Transport initialized, connection ready
2022-03-22 03:11:17.132 DEBUG 10 --- [er-nio-worker-1] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.233.68.68:9042 (1 connections needed, 1 successfully opened)
2022-03-22 03:11:17.132 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Session : Added connection pool for cassandra/10.233.68.68:9042
2022-03-22 03:11:17.629 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:17.638 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-2, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:11:17.640 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-2, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:11:17.640 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-2, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:11:17.641 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-2, inFlight=0, closed=true] closing connection
2022-03-22 03:11:17.642 DEBUG 10 --- [luster-worker-1] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:11:17.643 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 2000 milliseconds
2022-03-22 03:11:17.644 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 2000 ms
2022-03-22 03:11:18.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2022-03-22 03:11:18.224 DEBUG 10 --- [er-nio-worker-1] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.233.66.75:9042 (1 connections needed, 1 successfully opened)
2022-03-22 03:11:18.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Session : Added connection pool for /10.233.66.75:9042
2022-03-22 03:11:19.499 DEBUG 10 --- [ main] org.reflections.Reflections : going to scan these urls:
file:/opt/app/aai-traversal/resources/
2022-03-22 03:11:19.521 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/query/stored-queries.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.521 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/query/stored-queries.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.522 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v9/aai_oxm_v9.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.523 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v9/aai_oxm_v9.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.523 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v8/aai_oxm_v8.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.524 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v8/aai_oxm_v8.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.524 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v24/aai_oxm_v24.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.524 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v24/aai_oxm_v24.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.525 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v23/aai_oxm_v23.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.525 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v23/aai_oxm_v23.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.526 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v22/aai_oxm_v22.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.526 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v22/aai_oxm_v22.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.526 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v21/aai_oxm_v21.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.527 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v21/aai_oxm_v21.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.527 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v20/aai_oxm_v20.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.527 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v20/aai_oxm_v20.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.528 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v19/aai_oxm_v19.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.528 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v19/aai_oxm_v19.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.529 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v18/aai_oxm_v18.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.529 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v18/aai_oxm_v18.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.529 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v17/aai_oxm_v17.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.530 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v17/aai_oxm_v17.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.530 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v16/aai_oxm_v16.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.530 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v16/aai_oxm_v16.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.531 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v15/aai_oxm_v15.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.531 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v15/aai_oxm_v15.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.531 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v14/aai_oxm_v14.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.532 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v14/aai_oxm_v14.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.532 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v13/aai_oxm_v13.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.532 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v13/aai_oxm_v13.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.533 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v12/aai_oxm_v12.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.533 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v12/aai_oxm_v12.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.534 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v11/aai_oxm_v11.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.534 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v11/aai_oxm_v11.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.535 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v10/aai_oxm_v10.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.535 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v10/aai_oxm_v10.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.535 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v9/DbEdgeRules_v9.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.536 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v9/DbEdgeRules_v9.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.536 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v8/DbEdgeRules_v8.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.536 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v8/DbEdgeRules_v8.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.537 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.537 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.538 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_pnp_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.543 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_pnp_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.543 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_hpa_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.543 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_hpa_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.544 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_esr_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.544 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_esr_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.544 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_ccvpn_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.545 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_ccvpn_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.545 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_bbs_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.545 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v24/DbEdgeRules_bbs_v24.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.546 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.546 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.547 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_pnp_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.547 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_pnp_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.547 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_hpa_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.548 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_hpa_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.548 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_esr_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.548 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_esr_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.549 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_ccvpn_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.549 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_ccvpn_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.549 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_bbs_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.554 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_bbs_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.555 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.555 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.555 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_pnp_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.556 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_pnp_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.556 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_hpa_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.556 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_hpa_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.557 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_esr_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.557 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_esr_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.558 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_ccvpn_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.558 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_ccvpn_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.558 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_bbs_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.559 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_bbs_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.559 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.559 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.560 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_pnp_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.560 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_pnp_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.560 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_hpa_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.561 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_hpa_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.561 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_esr_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.561 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_esr_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.562 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_ccvpn_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.562 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_ccvpn_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.563 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_bbs_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.563 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_bbs_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.563 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.564 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.564 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_pnp_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.564 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_pnp_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.565 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_hpa_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.566 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_hpa_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.566 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_esr_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.566 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_esr_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.567 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_ccvpn_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.567 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_ccvpn_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.567 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_bbs_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.568 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_bbs_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.568 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.569 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.569 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_pnp_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.569 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_pnp_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.570 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_hpa_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.570 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_hpa_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.570 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_esr_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.571 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_esr_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.571 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_ccvpn_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.571 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_ccvpn_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.572 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_bbs_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.572 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_bbs_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.572 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.573 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.573 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_pnp_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.573 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_pnp_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.574 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_hpa_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.574 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_hpa_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.574 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_esr_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.575 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_esr_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.575 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_ccvpn_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.575 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_ccvpn_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.576 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_bbs_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.576 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_bbs_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.577 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.577 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.590 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_pnp_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.590 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_pnp_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.591 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_hpa_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.591 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_hpa_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.591 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_esr_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.591 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_esr_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.592 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_ccvpn_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.592 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_ccvpn_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.598 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_bbs_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.598 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_bbs_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.599 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.599 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.599 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_pnp_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.600 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_pnp_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.600 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_hpa_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.600 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_hpa_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.600 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_esr_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.601 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_esr_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.601 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_ccvpn_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.601 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_ccvpn_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.602 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_bbs_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.602 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_bbs_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.603 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.603 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.603 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_pnp_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.604 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_pnp_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.604 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_hpa_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.604 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_hpa_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.605 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_esr_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.605 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_esr_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.605 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_ccvpn_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.606 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_ccvpn_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.606 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.606 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.607 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_pnp_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.607 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_pnp_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.607 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_hpa_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.608 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_hpa_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.608 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_esr_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.608 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_esr_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.609 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_ccvpn_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.609 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_ccvpn_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.642 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.643 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.643 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_hpa_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.643 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_hpa_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.644 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_esr_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.644 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_esr_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.645 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.645 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.645 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_hpa_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.646 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_hpa_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.646 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_esr_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.646 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_esr_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.647 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v11/DbEdgeRules_v11.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.647 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v11/DbEdgeRules_v11.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.647 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:19.647 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v11/DbEdgeRules_esr_v11.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.648 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v11/DbEdgeRules_esr_v11.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.649 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v10/DbEdgeRules_v10.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.649 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v10/DbEdgeRules_v10.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.649 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file retired.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.650 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file retired.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.650 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file logback.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.650 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file logback.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.650 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file localhost-access-logback.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.651 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file localhost-access-logback.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.652 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/zone-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.652 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/zone-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.652 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.653 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.653 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpn-binding-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.654 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpn-binding-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.654 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpls-pe-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.654 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpls-pe-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.655 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpe-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.655 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpe-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.655 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/volume-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.655 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/volume-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.656 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/volume-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.656 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/volume-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.657 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vnfc-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.657 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vnfc-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.658 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vnf-image-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.685 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vnf-image-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.686 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-3, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:11:19.714 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-3, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:11:19.715 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-3, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:11:19.715 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-3, inFlight=0, closed=true] closing connection
2022-03-22 03:11:19.716 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 4000 milliseconds
2022-03-22 03:11:19.716 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 4000 ms
2022-03-22 03:11:19.718 DEBUG 10 --- [luster-worker-0] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:11:19.707 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vlan-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.719 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vlan-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.719 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/virtual-data-center-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.719 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/virtual-data-center-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.721 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vig-server-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.721 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vig-server-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.723 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vf-module-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.723 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vf-module-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.723 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vce-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.724 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vce-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.724 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/update-node-key-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.724 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/update-node-key-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.726 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tunnel-xconnect-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.726 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tunnel-xconnect-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.728 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tenant-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.728 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tenant-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.729 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tagged-inventory-item-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.729 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tagged-inventory-item-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.733 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/subnet-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.734 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/subnet-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.734 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/start-node-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.734 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/start-node-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.734 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/sriov-vf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.735 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/sriov-vf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.735 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/snapshot-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.735 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/snapshot-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.735 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/site-pair-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.736 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/site-pair-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.736 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/site-pair-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.736 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/site-pair-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.737 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-subscription-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.737 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-subscription-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.737 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.737 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.738 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-capability-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.738 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-capability-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.738 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.738 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.748 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/segmentation-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.748 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/segmentation-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.749 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/secondary-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.749 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/secondary-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.749 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/routing-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.750 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/routing-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.750 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/route-table-reference-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.750 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/route-table-reference-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.751 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/result-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.751 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/result-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.752 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/reserved-prop-names-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.752 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/reserved-prop-names-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.752 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/related-lookup-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.753 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/related-lookup-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.753 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/pserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.754 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/pserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.754 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/property-constraint-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.754 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/property-constraint-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.755 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/port-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.755 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/port-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.755 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/pnf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.756 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/pnf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.756 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/physical-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.757 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/physical-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.757 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/p-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.757 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/p-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.758 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/oam-network-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.758 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/oam-network-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.759 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/newvce-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.759 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/newvce-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.759 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/network-profile-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.760 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/network-profile-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.760 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/network-policy-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.760 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/network-policy-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.761 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/named-query-element-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.761 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/named-query-element-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.762 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/named-query-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.762 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/named-query-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.762 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/multicast-configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.763 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/multicast-configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.763 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-ver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.764 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-ver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.764 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-element-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.764 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-element-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.765 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-constraint-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.765 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-constraint-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.766 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.766 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.766 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/metadatum-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/metadatum-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/logical-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/logical-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.768 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/license-key-resource-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.768 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/license-key-resource-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.768 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/license-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.769 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/license-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.769 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/lag-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/lag-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/lag-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/lag-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-network-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-network-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-interface-ipv6-address-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-interface-ipv6-address-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-interface-ipv4-address-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-interface-ipv4-address-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/key-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/key-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ipsec-configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.775 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ipsec-configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.775 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/inventory-item-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/inventory-item-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/inventory-item-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/inventory-item-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/instance-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/instance-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/include-node-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/include-node-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/image-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/image-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.779 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/group-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.779 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/group-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/generic-vnf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/generic-vnf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/flavor-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/flavor-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/esr-system-info-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/esr-system-info-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.782 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/entitlement-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.782 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/entitlement-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/element-choice-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/element-choice-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/edge-prop-names-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/edge-prop-names-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/dvs-switch-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/dvs-switch-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/cvlan-tag-entry-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/cvlan-tag-entry-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/customer-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/customer-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ctag-pool-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ctag-pool-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ctag-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ctag-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/constrained-element-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/constrained-element-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/connector-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/connector-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/complex-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/complex-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/collection-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.791 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/collection-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.791 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/cloud-region-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.791 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/cloud-region-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/class-of-service-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/class-of-service-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/az-and-dvs-switches-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/az-and-dvs-switches-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/availability-zone-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/availability-zone-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/allotted-resource-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/allotted-resource-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/action-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/action-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/action-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/action-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/service-model-json/000-README in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/service-model-json/000-README in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/resource-model-json/000-README in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/resource-model-json/000-README in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/vnf-to-service-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/vnf-to-service-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/vnf-to-esr-system-info-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/vnf-to-esr-system-info-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.799 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getWlBundleId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.799 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getWlBundleId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.799 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getVserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.800 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getVserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.800 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getVnfVlanByCircuitId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.800 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getVnfVlanByCircuitId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.801 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getTenantInfoAtSvcSubscription-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.801 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getTenantInfoAtSvcSubscription-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getTenantInfoAtSvcInstance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getTenantInfoAtSvcInstance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getSvcSubscriberModelInfo-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getSvcSubscriberModelInfo-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.803 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getServiceInstanceSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.803 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getServiceInstanceSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.803 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getServiceInstanceModel-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.804 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getServiceInstanceModel-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.804 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getRouterRoadmTailSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.804 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getRouterRoadmTailSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.805 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getPinterfacePhysicalLinkBySvcInstId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.805 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getPinterfacePhysicalLinkBySvcInstId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.805 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getLogicalLinkByCloudRegionId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.806 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getLogicalLinkByCloudRegionId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.806 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getL3networkCloudRegionByNetworkRole-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.806 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getL3networkCloudRegionByNetworkRole-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.807 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology2-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.807 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology2-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.807 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology-1.1.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.808 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology-1.1.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.808 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.808 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.809 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVLogicalLinkByCircuitId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.809 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVLogicalLinkByCircuitId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.809 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVLogicalLink-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.810 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVLogicalLink-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.810 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVComplexLocationId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.810 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVComplexLocationId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.811 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVComplexHostname-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.811 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVComplexHostname-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.811 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.2.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.2.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.1.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.1.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComplexByPnfName-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComplexByPnfName-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getClfiRoadmTailSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getClfiRoadmTailSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/closed-loop-named-query-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/closed-loop-named-query-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/auth/realm.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.816 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/auth/realm.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.816 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/auth/aai_keystore in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.816 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/auth/aai_keystore in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/logging.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/logging.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/janusgraph-realtime.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/janusgraph-realtime.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/janusgraph-cached.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/janusgraph-cached.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.819 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/error.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.819 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/error.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.819 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/aaiconfig.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/aaiconfig.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/aaiEventDMaaPPublisher.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/aaiEventDMaaPPublisher.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file cadi.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file cadi.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file application.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file application.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file application-keycloak.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file application-keycloak.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.823 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/dsl/v2/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.823 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/dsl/v2/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/dsl/v1/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/dsl/v1/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/truststoreONAPall.jks in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/truststoreONAPall.jks in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/permissions.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/permissions.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.osaaf.location.props in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.osaaf.location.props in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.props in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.props in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.p12 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.828 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.p12 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.828 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.keyfile in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.828 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.keyfile in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/truststoreONAPall.jks in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/truststoreONAPall.jks in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.osaaf.location.props in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.osaaf.location.props in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.props in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.props in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.p12 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.p12 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.keyfile in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.832 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.keyfile in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.832 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/bath_config.csv in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2022-03-22 03:11:19.832 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/bath_config.csv in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2022-03-22 03:11:19.833 INFO 10 --- [ main] org.reflections.Reflections : Reflections took 332 ms to scan 1 urls, producing 0 keys and 0 values
2022-03-22 03:11:20.913 DEBUG 10 --- [ main] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=true] closing connection
2022-03-22 03:11:20.915 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-1, inFlight=0, closed=true] closed, remaining = 0
2022-03-22 03:11:20.916 DEBUG 10 --- [ main] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=true] has already terminated
2022-03-22 03:11:20.918 DEBUG 10 --- [ main] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=true] closing connection
2022-03-22 03:11:20.918 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=true] closed, remaining = 1
2022-03-22 03:11:20.935 DEBUG 10 --- [ main] com.datastax.driver.core.Cluster : Shutting down
2022-03-22 03:11:20.939 DEBUG 10 --- [ main] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=true] closing connection
2022-03-22 03:11:20.940 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=true] closed, remaining = 0
2022-03-22 03:11:22.994 DEBUG 10 --- [er-nio-worker-0] io.netty.buffer.PoolThreadCache : Freed 25 thread-local buffer(s) from thread: JanusGraph Cluster-nio-worker-0
2022-03-22 03:11:23.016 DEBUG 10 --- [er-nio-worker-1] io.netty.buffer.PoolThreadCache : Freed 20 thread-local buffer(s) from thread: JanusGraph Cluster-nio-worker-1
2022-03-22 03:11:23.056 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.USE_NATIVE_CLOCK is undefined, using default value true
2022-03-22 03:11:23.056 INFO 10 --- [ main] com.datastax.driver.core.ClockFactory : Using native clock to generate timestamps.
2022-03-22 03:11:23.057 DEBUG 10 --- [ main] com.datastax.driver.core.Cluster : Starting new cluster with contact points [cassandra/10.233.68.68:9042]
2022-03-22 03:11:23.062 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:23.084 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:11:24.115 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2022-03-22 03:11:24.116 DEBUG 10 --- [ main] c.d.driver.core.ControlConnection : [Control connection] Refreshing node list and token map
2022-03-22 03:11:24.157 DEBUG 10 --- [ main] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema
2022-03-22 03:11:24.541 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [Control connection] established to cassandra/10.233.68.68:9042
2022-03-22 03:11:24.542 INFO 10 --- [ main] c.d.d.c.p.DCAwareRoundRobinPolicy : Using data-center name 'Pod' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2022-03-22 03:11:24.542 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host /10.233.66.75:9042 added
2022-03-22 03:11:24.542 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host /10.233.71.64:9042 added
2022-03-22 03:11:24.542 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host cassandra/10.233.68.68:9042 added
2022-03-22 03:11:24.542 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:24.549 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:24.551 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] preparing to open 1 new connections, total = 2
2022-03-22 03:11:24.572 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-1, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:11:24.574 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-1, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:11:24.574 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preventing new connections for the next 1000 ms
2022-03-22 03:11:24.574 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-1, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:11:24.574 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:11:24.575 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:11:24.582 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-1, inFlight=0, closed=true] closing connection
2022-03-22 03:11:24.582 DEBUG 10 --- [luster-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host DOWN
2022-03-22 03:11:24.583 DEBUG 10 --- [luster-worker-0] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 03:11:24.584 WARN 10 --- [er-nio-worker-0] c.d.driver.core.HostConnectionPool : Error creating connection to /10.233.71.64:9042
com.datastax.driver.core.exceptions.TransportException: [/10.233.71.64:9042] Cannot connect
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:224)
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:194)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.71.64:9042
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 03:11:24.586 WARN 10 --- [er-nio-worker-0] com.datastax.driver.core.Session : Error creating pool to /10.233.71.64:9042
com.datastax.driver.core.exceptions.ConnectionException: [/10.233.71.64:9042] Pool was closed during initialization
at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173)
at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1773)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.google.common.util.concurrent.CollectionFuture$CollectionFutureRunningState.handleAllCompleted(CollectionFuture.java:76)
at com.google.common.util.concurrent.AggregateFuture$RunningState.processCompleted(AggregateFuture.java:255)
at com.google.common.util.concurrent.AggregateFuture$RunningState.decrementCountAndMaybeComplete(AggregateFuture.java:242)
at com.google.common.util.concurrent.AggregateFuture$RunningState.access$300(AggregateFuture.java:91)
at com.google.common.util.concurrent.AggregateFuture$RunningState$1.run(AggregateFuture.java:146)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.completeWithFuture(AbstractFuture.java:730)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:666)
at com.google.common.util.concurrent.Futures$AsyncCatchingFuture.doFallback(Futures.java:826)
at com.google.common.util.concurrent.Futures$AsyncCatchingFuture.doFallback(Futures.java:813)
at com.google.common.util.concurrent.Futures$AbstractCatchingFuture.run(Futures.java:789)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.completeWithFuture(AbstractFuture.java:730)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:666)
at com.google.common.util.concurrent.Futures$AsyncCatchingFuture.doFallback(Futures.java:826)
at com.google.common.util.concurrent.Futures$AsyncCatchingFuture.doFallback(Futures.java:813)
at com.google.common.util.concurrent.Futures$AbstractCatchingFuture.run(Futures.java:789)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1405)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:53)
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:222)
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:194)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 03:11:24.608 DEBUG 10 --- [luster-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] starting reconnection attempt
2022-03-22 03:11:25.583 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:25.603 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-2, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:11:25.603 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-2, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:11:25.604 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-2, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:11:25.604 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-2, inFlight=0, closed=true] closing connection
2022-03-22 03:11:25.605 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 2000 milliseconds
2022-03-22 03:11:25.606 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 2000 ms
2022-03-22 03:11:25.606 DEBUG 10 --- [luster-worker-1] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:11:26.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.233.68.68:9042] Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] Transport initialized, connection ready
2022-03-22 03:11:26.044 DEBUG 10 --- [er-nio-worker-1] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.233.68.68:9042 (1 connections needed, 1 successfully opened)
2022-03-22 03:11:26.044 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Session : Added connection pool for cassandra/10.233.68.68:9042
2022-03-22 03:11:26.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2022-03-22 03:11:26.408 DEBUG 10 --- [er-nio-worker-1] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.233.66.75:9042 (1 connections needed, 1 successfully opened)
2022-03-22 03:11:26.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Session : Added connection pool for /10.233.66.75:9042
2022-03-22 03:11:27.606 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:27.618 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-3, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:11:27.619 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-3, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:11:27.619 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-3, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:11:27.619 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-3, inFlight=0, closed=true] closing connection
2022-03-22 03:11:27.619 DEBUG 10 --- [luster-worker-0] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:11:27.620 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 4000 milliseconds
2022-03-22 03:11:27.620 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 4000 ms
2022-03-22 03:11:31.620 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:31.678 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-4, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:11:31.679 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-4, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:11:31.679 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-4, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:11:31.679 DEBUG 10 --- [luster-worker-1] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:11:31.679 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-4, inFlight=0, closed=true] closing connection
2022-03-22 03:11:31.680 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 8000 milliseconds
2022-03-22 03:11:31.680 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 8000 ms
2022-03-22 03:11:32.999 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Processing the translator
2022-03-22 03:11:33.013 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv11
2022-03-22 03:11:33.014 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:33.014 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v11, BODY=
2022-03-22 03:11:33.019 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:33.019 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:33.019 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:33.019 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:33.126 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:33.134 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:33.135 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:33.135 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv12
2022-03-22 03:11:33.136 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:33.137 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v12, BODY=
2022-03-22 03:11:33.140 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:33.140 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:33.140 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:33.140 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:33.271 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:33.286 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:33.287 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:33.287 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv13
2022-03-22 03:11:33.288 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:33.289 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v13, BODY=
2022-03-22 03:11:33.291 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:33.291 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:33.291 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:33.291 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:33.395 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:33.411 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:33.412 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:33.412 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv14
2022-03-22 03:11:33.413 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:33.413 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v14, BODY=
2022-03-22 03:11:33.416 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:33.416 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:33.416 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:33.416 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:33.518 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:33.559 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:33.560 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:33.561 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv15
2022-03-22 03:11:33.562 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:33.565 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v15, BODY=
2022-03-22 03:11:33.569 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:33.569 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:33.569 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:33.569 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:33.637 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:33.667 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:33.668 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:33.668 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv16
2022-03-22 03:11:33.669 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:33.669 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v16, BODY=
2022-03-22 03:11:33.690 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:33.690 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:33.690 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:33.690 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:33.784 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:33.821 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:33.825 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:33.830 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv17
2022-03-22 03:11:33.830 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:33.832 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v17, BODY=
2022-03-22 03:11:33.835 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:33.836 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:33.836 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:33.836 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:33.882 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:33.896 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:33.897 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:33.897 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv18
2022-03-22 03:11:33.898 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:33.899 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v18, BODY=
2022-03-22 03:11:33.904 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:33.904 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:33.904 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:33.904 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:33.966 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:33.987 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:33.988 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:33.988 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv19
2022-03-22 03:11:33.989 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:33.990 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v19, BODY=
2022-03-22 03:11:33.992 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:33.992 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:33.993 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:33.993 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:34.064 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:34.073 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:34.074 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:34.075 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv20
2022-03-22 03:11:34.075 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:34.076 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v20, BODY=
2022-03-22 03:11:34.086 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:34.086 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:34.086 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:34.087 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:34.579 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:34.613 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:34.613 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:34.614 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv21
2022-03-22 03:11:34.614 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:34.615 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v21, BODY=
2022-03-22 03:11:34.617 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:34.617 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:34.617 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:34.617 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:34.694 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:34.711 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:34.712 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:34.712 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv22
2022-03-22 03:11:34.713 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:34.713 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v22, BODY=
2022-03-22 03:11:34.716 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:34.716 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:34.716 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:34.716 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:34.801 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:34.823 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:34.824 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:34.825 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv23
2022-03-22 03:11:34.832 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:34.834 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v23, BODY=
2022-03-22 03:11:34.842 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:34.844 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:34.844 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:34.844 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:34.897 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:34.915 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:34.916 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:34.916 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv24
2022-03-22 03:11:34.916 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:34.917 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v24, BODY=
2022-03-22 03:11:34.919 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:34.919 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:34.919 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:34.919 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:34.984 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:35.017 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2022-03-22 03:11:35.018 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2022-03-22 03:11:35.403 INFO 10 --- [ main] o.o.a.rest.search.SchemaServiceCQConfig : Calling the SchemaService to retrieve stored queries
2022-03-22 03:11:35.404 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2022-03-22 03:11:35.405 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/stored-queries, BODY=
2022-03-22 03:11:35.407 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2022-03-22 03:11:35.408 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2022-03-22 03:11:35.408 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2022-03-22 03:11:35.408 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2022-03-22 03:11:35.471 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2022-03-22 03:11:38.365 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Found javax.persistence.Persistence on classpath containing 'getPersistenceUtil'. Assuming JPA 2 environment. Trying to instantiate JPA aware TraversableResolver
2022-03-22 03:11:38.365 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Instantiated JPA aware TraversableResolver of type org.hibernate.validator.internal.engine.resolver.JPATraversableResolver.
2022-03-22 03:11:38.403 DEBUG 10 --- [ main] .h.v.m.ResourceBundleMessageInterpolator : Loaded expression factory via original TCCL
2022-03-22 03:11:38.408 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Found javax.persistence.Persistence on classpath containing 'getPersistenceUtil'. Assuming JPA 2 environment. Trying to instantiate JPA aware TraversableResolver
2022-03-22 03:11:38.409 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Instantiated JPA aware TraversableResolver of type org.hibernate.validator.internal.engine.resolver.JPATraversableResolver.
2022-03-22 03:11:38.412 DEBUG 10 --- [ main] o.h.v.internal.engine.ConfigurationImpl : Setting custom MessageInterpolator of type org.springframework.validation.beanvalidation.LocaleContextMessageInterpolator
2022-03-22 03:11:38.433 DEBUG 10 --- [ main] o.h.v.internal.engine.ConfigurationImpl : Setting custom ConstraintValidatorFactory of type org.springframework.validation.beanvalidation.SpringConstraintValidatorFactory
2022-03-22 03:11:38.435 DEBUG 10 --- [ main] o.h.v.internal.engine.ConfigurationImpl : Setting custom ParameterNameProvider of type org.springframework.validation.beanvalidation.LocalValidatorFactoryBean$1
2022-03-22 03:11:38.443 DEBUG 10 --- [ main] o.h.v.i.xml.config.ValidationXmlParser : Trying to load META-INF/validation.xml for XML based Validator configuration.
2022-03-22 03:11:38.443 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via user class loader
2022-03-22 03:11:38.445 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via TCCL
2022-03-22 03:11:38.446 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via Hibernate Validator's class loader
2022-03-22 03:11:38.448 DEBUG 10 --- [ main] o.h.v.i.xml.config.ValidationXmlParser : No META-INF/validation.xml found. Using annotation based configuration only.
2022-03-22 03:11:38.504 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.springframework.validation.beanvalidation.LocaleContextMessageInterpolator as ValidatorFactory-scoped message interpolator.
2022-03-22 03:11:38.505 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.resolver.JPATraversableResolver as ValidatorFactory-scoped traversable resolver.
2022-03-22 03:11:38.505 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.util.ExecutableParameterNameProvider as ValidatorFactory-scoped parameter name provider.
2022-03-22 03:11:38.505 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.DefaultClockProvider as ValidatorFactory-scoped clock provider.
2022-03-22 03:11:38.505 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.scripting.DefaultScriptEvaluatorFactory as ValidatorFactory-scoped script evaluator factory.
2022-03-22 03:11:39.698 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:39.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-5, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:11:39.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-5, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:11:39.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-5, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:11:39.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-5, inFlight=0, closed=true] closing connection
2022-03-22 03:11:39.729 DEBUG 10 --- [luster-worker-0] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:11:39.739 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 16000 milliseconds
2022-03-22 03:11:39.739 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 16000 ms
2022-03-22 03:11:40.444 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.USE_NATIVE_CLOCK is undefined, using default value true
2022-03-22 03:11:40.444 INFO 10 --- [ main] com.datastax.driver.core.ClockFactory : Using native clock to generate timestamps.
2022-03-22 03:11:40.480 WARN 10 --- [ main] .b.a.g.t.GroovyTemplateAutoConfiguration : Cannot find template location: classpath:/templates/ (please add some templates, check your Groovy configuration, or set spring.groovy.template.check-template-location=false)
2022-03-22 03:11:46.898 DEBUG 10 --- [ main] i.m.s.web.servlet.WebMvcMetricsFilter : Filter 'webMetricsFilter' configured for use
2022-03-22 03:11:52.285 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Found javax.persistence.Persistence on classpath containing 'getPersistenceUtil'. Assuming JPA 2 environment. Trying to instantiate JPA aware TraversableResolver
2022-03-22 03:11:52.285 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Instantiated JPA aware TraversableResolver of type org.hibernate.validator.internal.engine.resolver.JPATraversableResolver.
2022-03-22 03:11:52.289 DEBUG 10 --- [ main] o.h.v.i.xml.config.ValidationXmlParser : Trying to load META-INF/validation.xml for XML based Validator configuration.
2022-03-22 03:11:52.289 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via TCCL
2022-03-22 03:11:52.290 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via Hibernate Validator's class loader
2022-03-22 03:11:52.291 DEBUG 10 --- [ main] o.h.v.i.xml.config.ValidationXmlParser : No META-INF/validation.xml found. Using annotation based configuration only.
2022-03-22 03:11:52.329 DEBUG 10 --- [ main] .h.v.m.ResourceBundleMessageInterpolator : Loaded expression factory via original TCCL
2022-03-22 03:11:52.330 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.messageinterpolation.ResourceBundleMessageInterpolator as ValidatorFactory-scoped message interpolator.
2022-03-22 03:11:52.330 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.resolver.JPATraversableResolver as ValidatorFactory-scoped traversable resolver.
2022-03-22 03:11:52.330 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.util.ExecutableParameterNameProvider as ValidatorFactory-scoped parameter name provider.
2022-03-22 03:11:52.330 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.DefaultClockProvider as ValidatorFactory-scoped clock provider.
2022-03-22 03:11:52.330 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.scripting.DefaultScriptEvaluatorFactory as ValidatorFactory-scoped script evaluator factory.
2022-03-22 03:11:52.535 WARN 10 --- [ main] org.glassfish.jersey.internal.Errors : The following warnings have been detected: WARNING: The (sub)resource method getC2Qgremlin in org.onap.aai.rest.CQ2Gremlin contains empty path annotation.
WARNING: The (sub)resource method getC2Qgremlin in org.onap.aai.rest.CQ2GremlinTest contains empty path annotation.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.onap.aai.rest.RecentAPIConsumer.getRecentData(java.lang.String,java.lang.String,java.lang.String,javax.ws.rs.core.HttpHeaders,javax.servlet.http.HttpServletRequest,javax.ws.rs.core.UriInfo), should not consume any entity.
2022-03-22 03:11:52.562 DEBUG 10 --- [ main] org.onap.aai.config.ErrorHandler : Filter 'errorHandler' configured for use
2022-03-22 03:11:53.792 INFO 10 --- [ main] org.onap.aai.TraversalApp : Application 'aai-traversal' is running on 8446!
2022-03-22 03:11:53.794 DEBUG 10 --- [ main] org.onap.aai.TraversalApp : Traversal MicroService Started
Traversal Microservice Started
2022-03-22 03:11:54.377 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:11:54.416 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:11:55.750 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:11:55.759 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-6, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:11:55.760 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-6, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:11:55.760 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-6, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:11:55.760 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-6, inFlight=0, closed=true] closing connection
2022-03-22 03:11:55.761 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 03:11:55.761 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 32000 ms
2022-03-22 03:11:55.762 DEBUG 10 --- [luster-worker-1] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:12:24.417 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:12:24.505 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:12:27.762 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:12:27.795 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-7, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:12:27.795 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-7, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:12:27.796 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-7, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:12:27.797 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-7, inFlight=0, closed=true] closing connection
2022-03-22 03:12:27.798 DEBUG 10 --- [luster-worker-2] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:12:27.799 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 03:12:27.800 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 64000 ms
2022-03-22 03:12:54.506 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:12:54.603 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:13:24.604 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:13:24.612 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:13:31.800 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:13:31.862 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-8, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:13:31.863 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-8, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:13:31.863 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-8, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:13:31.864 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-8, inFlight=0, closed=true] closing connection
2022-03-22 03:13:31.867 DEBUG 10 --- [luster-worker-3] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:13:31.872 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 03:13:31.872 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 128000 ms
2022-03-22 03:13:54.634 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:13:54.725 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:14:24.736 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:14:24.927 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:14:54.928 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:14:55.015 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:15:16.705 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UP /10.233.71.64:9042, scheduling delivery
2022-03-22 03:15:16.726 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] received event UP
2022-03-22 03:15:17.758 DEBUG 10 --- [luster-worker-4] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.233.71.64:9042
2022-03-22 03:15:17.775 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host UP
2022-03-22 03:15:17.775 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2022-03-22 03:15:17.776 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.233.71.64:9042
2022-03-22 03:15:17.776 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:15:17.793 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-9, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:15:19.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:15:19.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:15:38.910 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-9, inFlight=0, closed=false] because: [/10.233.71.64:9042] Operation timed out
2022-03-22 03:15:38.911 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-9, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:15:38.912 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-9, inFlight=0, closed=true] closing connection
2022-03-22 03:15:38.912 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-9, inFlight=0, closed=true]: there are still pending requests
2022-03-22 03:15:38.915 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:15:38.937 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-10, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:15:47.783 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:15:47.834 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:15:52.190 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-10, inFlight=0, closed=false] because: [/10.233.71.64:9042] Operation timed out
2022-03-22 03:15:52.191 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-10, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:15:52.191 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-10, inFlight=0, closed=true] closing connection
2022-03-22 03:15:52.191 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-10, inFlight=0, closed=true]: there are still pending requests
2022-03-22 03:15:52.192 WARN 10 --- [ter-timeouter-0] c.d.driver.core.HostConnectionPool : Error creating connection to /10.233.71.64:9042
com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.233.71.64:9042] Operation timed out
at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1496)
at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1575)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 03:15:52.193 DEBUG 10 --- [ter-timeouter-0] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.233.71.64:9042 (1 connections needed, 0 successfully opened)
2022-03-22 03:15:52.193 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Session : Added connection pool for /10.233.71.64:9042
2022-03-22 03:15:52.201 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host DOWN
2022-03-22 03:15:52.201 DEBUG 10 --- [luster-worker-5] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 03:15:52.202 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] starting reconnection attempt
2022-03-22 03:15:52.202 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:15:53.202 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:15:53.207 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-11, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:16:06.988 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-11, inFlight=0, closed=false] because: [/10.233.71.64:9042] Operation timed out
2022-03-22 03:16:06.988 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-11, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:16:06.988 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-11, inFlight=0, closed=true] closing connection
2022-03-22 03:16:06.988 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:16:06.988 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-11, inFlight=0, closed=true]: there are still pending requests
2022-03-22 03:16:06.989 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Operation timed out), scheduling retry in 2000 milliseconds
2022-03-22 03:16:06.989 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 2000 ms
2022-03-22 03:16:08.096 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-9, inFlight=0, closed=true] has already terminated
2022-03-22 03:16:08.990 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:16:09.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-12, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:16:11.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:16:11.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:16:17.835 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:16:17.935 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:16:21.890 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-12, inFlight=0, closed=false] because: [/10.233.71.64:9042] Operation timed out
2022-03-22 03:16:21.892 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-12, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:16:21.893 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:16:21.893 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-12, inFlight=0, closed=true] closing connection
2022-03-22 03:16:21.894 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-12, inFlight=0, closed=true]: there are still pending requests
2022-03-22 03:16:21.895 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Operation timed out), scheduling retry in 4000 milliseconds
2022-03-22 03:16:21.895 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 4000 ms
2022-03-22 03:16:23.097 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-10, inFlight=0, closed=true] has already terminated
2022-03-22 03:16:25.896 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:16:25.919 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-13, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:16:38.098 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-11, inFlight=0, closed=true] has already terminated
2022-03-22 03:16:38.188 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-13, inFlight=0, closed=false] because: [/10.233.71.64:9042] Operation timed out
2022-03-22 03:16:38.188 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-13, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:16:38.189 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-13, inFlight=0, closed=true] closing connection
2022-03-22 03:16:38.189 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-13, inFlight=0, closed=true]: there are still pending requests
2022-03-22 03:16:38.190 DEBUG 10 --- [luster-worker-6] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:16:38.190 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Operation timed out), scheduling retry in 8000 milliseconds
2022-03-22 03:16:38.191 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 8000 ms
2022-03-22 03:16:46.191 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:16:46.207 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-14, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:16:47.936 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:16:48.101 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:16:53.099 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-12, inFlight=0, closed=true] has already terminated
2022-03-22 03:16:58.387 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-14, inFlight=0, closed=false] because: [/10.233.71.64:9042] Operation timed out
2022-03-22 03:16:58.387 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-14, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:16:58.389 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-14, inFlight=0, closed=true] closing connection
2022-03-22 03:16:58.389 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-14, inFlight=0, closed=true]: there are still pending requests
2022-03-22 03:16:58.389 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:16:58.389 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Operation timed out), scheduling retry in 16000 milliseconds
2022-03-22 03:16:58.390 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 16000 ms
2022-03-22 03:17:08.100 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-13, inFlight=0, closed=true] has already terminated
2022-03-22 03:17:14.390 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:17:14.406 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:17:18.105 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:17:18.124 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:17:20.679 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-15, inFlight=0, closed=false] Transport initialized, connection ready
2022-03-22 03:17:20.680 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.233.71.64:9042
2022-03-22 03:17:20.720 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Successful reconnection to /10.233.71.64:9042, setting host UP
2022-03-22 03:17:20.720 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host UP
2022-03-22 03:17:20.720 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2022-03-22 03:17:20.720 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.233.71.64:9042
2022-03-22 03:17:23.101 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-14, inFlight=0, closed=true] has already terminated
2022-03-22 03:17:25.287 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 0 new connections, total = 1
2022-03-22 03:17:25.287 DEBUG 10 --- [-reconnection-1] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.233.71.64:9042 (1 connections needed, 1 successfully opened)
2022-03-22 03:17:25.287 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Session : Added connection pool for /10.233.71.64:9042
2022-03-22 03:17:25.288 DEBUG 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2022-03-22 03:17:50.721 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:17:50.823 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:18:04.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:18:04.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:18:20.825 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:18:20.901 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:18:22.909 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:18:25.080 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:18:50.902 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:18:50.912 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:18:58.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:18:58.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:19:13.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:19:13.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:19:20.914 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:19:21.001 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:19:51.003 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:19:51.009 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:20:16.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:20:16.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:20:21.032 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:20:21.102 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:20:51.105 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:20:51.113 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:21:21.121 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:21:21.204 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:21:51.205 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:21:51.210 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:21:58.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:21:58.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:22:21.234 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:22:21.316 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:22:28.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:22:28.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:22:51.319 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:22:51.407 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:22:53.083 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:22:53.186 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:23:21.408 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:23:21.413 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:23:37.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:23:37.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:23:51.438 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:23:51.519 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:23:57.731 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:23:58.581 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:24:13.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:24:13.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:24:21.521 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:24:21.617 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:24:33.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:24:33.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:24:51.618 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:24:51.642 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:25:16.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:25:16.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:25:21.648 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:25:21.759 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:25:30.437 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.233.71.64:9042, scheduling delivery
2022-03-22 03:25:30.438 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] received event DOWN
2022-03-22 03:25:30.676 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=1, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 03:25:30.678 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-15, inFlight=1, closed=false] because: [/10.233.71.64:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2022-03-22 03:25:30.678 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preventing new connections for the next 1000 ms
2022-03-22 03:25:30.679 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-15, inFlight=1, closed=false] failed, remaining = 0
2022-03-22 03:25:30.681 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=1, closed=true] closing connection
2022-03-22 03:25:30.681 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-15, inFlight=1, closed=true]: there are still pending requests
2022-03-22 03:25:30.682 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host DOWN
2022-03-22 03:25:30.683 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.RequestHandler : onException triggered but the response was completed by another thread, cancelling (retryCount = 0, queryState = QueryState(count=0, inProgress=false, cancelled=false), queryStateRef = QueryState(count=0, inProgress=false, cancelled=false))
2022-03-22 03:25:30.689 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.RequestHandler : [1915292973-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67f39ed1 at consistency LOCAL_QUORUM
2022-03-22 03:25:30.690 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.RequestHandler : [1915292973-0] Error querying /10.233.71.64:9042 : com.datastax.driver.core.exceptions.TransportException: [/10.233.71.64:9042] Connection has been closed
2022-03-22 03:25:30.693 DEBUG 10 --- [luster-worker-8] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 03:25:30.693 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] starting reconnection attempt
2022-03-22 03:25:31.445 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:25:31.694 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:25:31.736 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-16, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:25:31.737 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-16, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:25:31.737 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-16, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:25:31.737 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-16, inFlight=0, closed=true] closing connection
2022-03-22 03:25:31.737 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:25:31.746 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 2000 milliseconds
2022-03-22 03:25:31.747 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 2000 ms
2022-03-22 03:25:33.747 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:25:33.752 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-17, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:25:33.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-17, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:25:33.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-17, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:25:33.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-17, inFlight=0, closed=true] closing connection
2022-03-22 03:25:33.753 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:25:33.754 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 4000 milliseconds
2022-03-22 03:25:33.754 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 4000 ms
2022-03-22 03:25:37.755 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:25:37.762 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-18, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:25:37.762 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-18, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:25:37.762 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-18, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:25:37.763 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-18, inFlight=0, closed=true] closing connection
2022-03-22 03:25:37.763 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 8000 milliseconds
2022-03-22 03:25:37.763 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 8000 ms
2022-03-22 03:25:37.765 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:25:45.763 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:25:45.777 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-19, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:25:45.777 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-19, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:25:45.777 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-19, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:25:45.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-19, inFlight=0, closed=true] closing connection
2022-03-22 03:25:45.778 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 16000 milliseconds
2022-03-22 03:25:45.778 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 16000 ms
2022-03-22 03:25:45.781 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:26:00.448 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:26:00.505 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:26:01.781 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:26:01.799 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-20, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:26:01.799 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-20, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:26:01.800 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-20, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:26:01.800 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-20, inFlight=0, closed=true] closing connection
2022-03-22 03:26:01.800 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 03:26:01.801 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 32000 ms
2022-03-22 03:26:01.802 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:26:08.149 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-15, inFlight=0, closed=true] has already terminated
2022-03-22 03:26:30.507 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:26:30.526 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:26:33.801 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:26:33.805 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-21, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:26:33.806 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-21, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:26:33.806 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-21, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:26:33.807 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-21, inFlight=0, closed=true] closing connection
2022-03-22 03:26:33.807 DEBUG 10 --- [uster-worker-10] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:26:33.807 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 03:26:33.808 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 64000 ms
2022-03-22 03:27:00.535 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:27:00.622 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:27:30.623 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:27:30.626 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:27:37.808 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:27:37.811 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-22, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 03:27:37.812 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-22, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 03:27:37.812 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-22, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 03:27:37.813 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-22, inFlight=0, closed=true] closing connection
2022-03-22 03:27:37.815 DEBUG 10 --- [uster-worker-11] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 03:27:37.815 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 03:27:37.818 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 128000 ms
2022-03-22 03:27:53.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:27:53.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:28:00.642 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:28:00.646 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:28:30.673 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:28:30.697 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:28:34.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:28:34.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:28:53.512 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UP /10.233.71.64:9042, scheduling delivery
2022-03-22 03:28:53.512 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] received event UP
2022-03-22 03:28:54.514 DEBUG 10 --- [uster-worker-12] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.233.71.64:9042
2022-03-22 03:28:54.609 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host UP
2022-03-22 03:28:54.609 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2022-03-22 03:28:54.609 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.233.71.64:9042
2022-03-22 03:28:54.610 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 03:28:54.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 03:28:59.087 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] Transport initialized, connection ready
2022-03-22 03:29:00.207 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 0 new connections, total = 1
2022-03-22 03:29:00.207 DEBUG 10 --- [uster-worker-12] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.233.71.64:9042 (1 connections needed, 1 successfully opened)
2022-03-22 03:29:00.208 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Session : Added connection pool for /10.233.71.64:9042
2022-03-22 03:29:24.616 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:29:24.631 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:29:30.881 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:29:30.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:29:54.645 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:29:54.707 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:29:56.900 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:29:57.397 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:30:17.433 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:30:17.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:30:24.709 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:30:24.821 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:30:53.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:30:53.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:30:54.822 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:30:54.913 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:31:17.526 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE sdcartifact, scheduling delivery
2022-03-22 03:31:18.546 DEBUG 10 --- [uster-worker-13] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcartifact
2022-03-22 03:31:18.711 DEBUG 10 --- [uster-worker-13] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdcartifact.
2022-03-22 03:31:18.734 DEBUG 10 --- [uster-worker-13] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdcartifact completed in 22 milliseconds
2022-03-22 03:31:32.615 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcartifact.resources, scheduling delivery
2022-03-22 03:31:33.619 DEBUG 10 --- [uster-worker-14] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcartifact.resources (TABLE)
2022-03-22 03:31:41.682 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcartifact.sdcschemafiles, scheduling delivery
2022-03-22 03:31:42.683 DEBUG 10 --- [uster-worker-13] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcartifact.sdcschemafiles (TABLE)
2022-03-22 03:31:47.619 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcartifact.tosca_import_by_model, scheduling delivery
2022-03-22 03:31:48.619 DEBUG 10 --- [uster-worker-14] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcartifact.tosca_import_by_model (TABLE)
2022-03-22 03:31:57.215 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcartifact.tosca_import_by_model, scheduling delivery
2022-03-22 03:31:58.217 DEBUG 10 --- [uster-worker-13] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcartifact.tosca_import_by_model (TABLE)
2022-03-22 03:32:01.003 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE sdcrepository, scheduling delivery
2022-03-22 03:32:02.004 DEBUG 10 --- [uster-worker-14] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcrepository
2022-03-22 03:32:02.024 DEBUG 10 --- [uster-worker-14] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdcartifact.
2022-03-22 03:32:02.048 DEBUG 10 --- [uster-worker-14] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdcartifact completed in 24 milliseconds
2022-03-22 03:32:05.312 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcrepository.migrationtasks, scheduling delivery
2022-03-22 03:32:06.315 DEBUG 10 --- [uster-worker-13] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcrepository.migrationtasks (TABLE)
2022-03-22 03:32:11.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:32:12.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:32:15.508 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcrepository.operationalenvironment, scheduling delivery
2022-03-22 03:32:16.510 DEBUG 10 --- [uster-worker-14] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcrepository.operationalenvironment (TABLE)
2022-03-22 03:32:32.327 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcrepository.operationalenvironment, scheduling delivery
2022-03-22 03:32:33.328 DEBUG 10 --- [uster-worker-13] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcrepository.operationalenvironment (TABLE)
2022-03-22 03:32:37.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:32:37.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:32:47.397 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:32:47.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:33:03.433 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:33:03.511 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:33:07.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:33:07.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:33:09.078 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcrepository.featuretogglestate, scheduling delivery
2022-03-22 03:33:10.088 DEBUG 10 --- [uster-worker-15] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcrepository.featuretogglestate (TABLE)
2022-03-22 03:33:13.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:33:13.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:33:25.811 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE sdcaudit, scheduling delivery
2022-03-22 03:33:26.814 DEBUG 10 --- [uster-worker-16] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit
2022-03-22 03:33:26.920 DEBUG 10 --- [uster-worker-16] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdcartifact.
2022-03-22 03:33:26.937 DEBUG 10 --- [uster-worker-16] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdcartifact completed in 17 milliseconds
2022-03-22 03:33:39.205 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.useradminevent, scheduling delivery
2022-03-22 03:33:40.208 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.useradminevent (TABLE)
2022-03-22 03:33:58.709 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.useradminevent, scheduling delivery
2022-03-22 03:33:59.715 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.useradminevent (TABLE)
2022-03-22 03:34:02.806 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.useraccessevent, scheduling delivery
2022-03-22 03:34:03.807 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.useraccessevent (TABLE)
2022-03-22 03:34:07.044 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.useraccessevent, scheduling delivery
2022-03-22 03:34:08.045 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.useraccessevent (TABLE)
2022-03-22 03:34:12.222 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.resourceadminevent, scheduling delivery
2022-03-22 03:34:13.224 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.resourceadminevent (TABLE)
2022-03-22 03:34:26.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.resourceadminevent, scheduling delivery
2022-03-22 03:34:27.504 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.resourceadminevent (TABLE)
2022-03-22 03:34:31.218 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.resourceadminevent, scheduling delivery
2022-03-22 03:34:31.376 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:34:31.381 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:34:32.219 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.resourceadminevent (TABLE)
2022-03-22 03:34:39.116 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.resourceadminevent, scheduling delivery
2022-03-22 03:34:40.116 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.resourceadminevent (TABLE)
2022-03-22 03:34:46.125 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.resourceadminevent, scheduling delivery
2022-03-22 03:34:47.126 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.resourceadminevent (TABLE)
2022-03-22 03:34:55.637 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.resourceadminevent, scheduling delivery
2022-03-22 03:34:56.639 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.resourceadminevent (TABLE)
2022-03-22 03:35:01.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:35:02.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:35:02.206 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.resourceadminevent, scheduling delivery
2022-03-22 03:35:03.207 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.resourceadminevent (TABLE)
2022-03-22 03:35:11.439 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.resourceadminevent, scheduling delivery
2022-03-22 03:35:12.445 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.resourceadminevent (TABLE)
2022-03-22 03:35:27.006 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.distributiondownloadevent, scheduling delivery
2022-03-22 03:35:28.007 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributiondownloadevent (TABLE)
2022-03-22 03:35:41.176 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributiondownloadevent, scheduling delivery
2022-03-22 03:35:42.180 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributiondownloadevent (TABLE)
2022-03-22 03:35:42.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:35:42.881 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:35:46.173 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.distributionengineevent, scheduling delivery
2022-03-22 03:35:47.176 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionengineevent (TABLE)
2022-03-22 03:35:58.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:35:58.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:36:07.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributionengineevent, scheduling delivery
2022-03-22 03:36:08.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:36:08.503 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionengineevent (TABLE)
2022-03-22 03:36:08.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:36:11.437 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributionengineevent, scheduling delivery
2022-03-22 03:36:12.441 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionengineevent (TABLE)
2022-03-22 03:36:28.314 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.distributionnotificationevent, scheduling delivery
2022-03-22 03:36:29.316 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionnotificationevent (TABLE)
2022-03-22 03:36:33.985 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:36:33.998 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:36:34.617 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributionnotificationevent, scheduling delivery
2022-03-22 03:36:35.618 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionnotificationevent (TABLE)
2022-03-22 03:36:39.529 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributionnotificationevent, scheduling delivery
2022-03-22 03:36:40.530 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionnotificationevent (TABLE)
2022-03-22 03:37:01.508 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributionnotificationevent, scheduling delivery
2022-03-22 03:37:02.509 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionnotificationevent (TABLE)
2022-03-22 03:37:04.330 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.distributionstatusevent, scheduling delivery
2022-03-22 03:37:05.331 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionstatusevent (TABLE)
2022-03-22 03:37:07.633 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributionstatusevent, scheduling delivery
2022-03-22 03:37:08.634 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionstatusevent (TABLE)
2022-03-22 03:37:14.586 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributionstatusevent, scheduling delivery
2022-03-22 03:37:15.587 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributionstatusevent (TABLE)
2022-03-22 03:37:20.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:37:20.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:37:34.607 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.distributiondeployevent, scheduling delivery
2022-03-22 03:37:35.609 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributiondeployevent (TABLE)
2022-03-22 03:37:39.198 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributiondeployevent, scheduling delivery
2022-03-22 03:37:40.199 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributiondeployevent (TABLE)
2022-03-22 03:37:41.081 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:37:41.086 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:37:50.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:37:50.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:37:50.610 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributiondeployevent, scheduling delivery
2022-03-22 03:37:51.611 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributiondeployevent (TABLE)
2022-03-22 03:38:02.915 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributiondeployevent, scheduling delivery
2022-03-22 03:38:03.915 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributiondeployevent (TABLE)
2022-03-22 03:38:08.199 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.distributiondeployevent, scheduling delivery
2022-03-22 03:38:09.201 DEBUG 10 --- [uster-worker-17] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.distributiondeployevent (TABLE)
2022-03-22 03:38:11.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:38:11.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:38:34.516 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.auditinggetuebclusterevent, scheduling delivery
2022-03-22 03:38:35.524 DEBUG 10 --- [uster-worker-19] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.auditinggetuebclusterevent (TABLE)
2022-03-22 03:38:43.008 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.auditinggetuebclusterevent, scheduling delivery
2022-03-22 03:38:44.011 DEBUG 10 --- [uster-worker-20] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.auditinggetuebclusterevent (TABLE)
2022-03-22 03:39:03.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:39:03.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:39:10.355 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.authevent, scheduling delivery
2022-03-22 03:39:11.359 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.authevent (TABLE)
2022-03-22 03:39:19.519 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.authevent, scheduling delivery
2022-03-22 03:39:20.521 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.authevent (TABLE)
2022-03-22 03:39:30.113 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.authevent, scheduling delivery
2022-03-22 03:39:31.115 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.authevent (TABLE)
2022-03-22 03:39:35.639 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.categoryevent, scheduling delivery
2022-03-22 03:39:36.640 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.categoryevent (TABLE)
2022-03-22 03:39:40.636 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.categoryevent, scheduling delivery
2022-03-22 03:39:41.637 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.categoryevent (TABLE)
2022-03-22 03:39:45.359 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.getuserslistevent, scheduling delivery
2022-03-22 03:39:46.361 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.getuserslistevent (TABLE)
2022-03-22 03:39:58.926 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.getuserslistevent, scheduling delivery
2022-03-22 03:39:59.931 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.getuserslistevent (TABLE)
2022-03-22 03:40:02.413 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.getcategoryhierarchyevent, scheduling delivery
2022-03-22 03:40:03.414 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.getcategoryhierarchyevent (TABLE)
2022-03-22 03:40:04.702 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.getcategoryhierarchyevent, scheduling delivery
2022-03-22 03:40:05.081 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:40:05.135 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:40:05.704 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.getcategoryhierarchyevent (TABLE)
2022-03-22 03:40:09.755 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.externalapievent, scheduling delivery
2022-03-22 03:40:10.757 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.externalapievent (TABLE)
2022-03-22 03:40:14.331 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.externalapievent, scheduling delivery
2022-03-22 03:40:15.332 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.externalapievent (TABLE)
2022-03-22 03:40:23.013 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.externalapievent, scheduling delivery
2022-03-22 03:40:24.014 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.externalapievent (TABLE)
2022-03-22 03:40:30.421 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE sdcaudit.externalapievent, scheduling delivery
2022-03-22 03:40:31.424 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.externalapievent (TABLE)
2022-03-22 03:40:36.326 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdcaudit.ecompopenvironmentevent, scheduling delivery
2022-03-22 03:40:37.327 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdcaudit.ecompopenvironmentevent (TABLE)
2022-03-22 03:40:55.415 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE sdctitan, scheduling delivery
2022-03-22 03:40:56.416 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan
2022-03-22 03:40:56.510 DEBUG 10 --- [uster-worker-21] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2022-03-22 03:40:56.522 DEBUG 10 --- [uster-worker-21] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 11 milliseconds
2022-03-22 03:40:59.419 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdctitan.system_properties, scheduling delivery
2022-03-22 03:41:00.419 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan.system_properties (TABLE)
2022-03-22 03:41:06.084 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:41:06.090 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:41:10.513 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdctitan.janusgraph_ids, scheduling delivery
2022-03-22 03:41:11.514 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan.janusgraph_ids (TABLE)
2022-03-22 03:41:13.860 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdctitan.edgestore, scheduling delivery
2022-03-22 03:41:14.861 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan.edgestore (TABLE)
2022-03-22 03:41:17.909 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdctitan.edgestore_lock_, scheduling delivery
2022-03-22 03:41:18.910 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan.edgestore_lock_ (TABLE)
2022-03-22 03:41:23.626 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdctitan.graphindex, scheduling delivery
2022-03-22 03:41:24.627 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan.graphindex (TABLE)
2022-03-22 03:41:29.321 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdctitan.graphindex_lock_, scheduling delivery
2022-03-22 03:41:30.322 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan.graphindex_lock_ (TABLE)
2022-03-22 03:41:31.952 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdctitan.txlog, scheduling delivery
2022-03-22 03:41:32.952 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan.txlog (TABLE)
2022-03-22 03:41:36.382 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:41:36.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:41:36.585 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdctitan.systemlog, scheduling delivery
2022-03-22 03:41:37.592 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan.systemlog (TABLE)
2022-03-22 03:41:44.813 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE sdctitan.system_properties_lock_, scheduling delivery
2022-03-22 03:41:45.814 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for sdctitan.system_properties_lock_ (TABLE)
2022-03-22 03:42:06.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:42:06.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:42:15.863 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:42:15.867 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:42:37.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:42:37.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:42:45.872 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:42:45.879 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:43:02.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:43:02.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:43:07.337 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:43:07.347 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:43:15.901 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:43:15.914 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:43:27.991 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:43:28.085 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:43:38.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:43:38.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:43:45.929 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:43:45.932 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:44:08.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:44:08.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:44:15.959 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:44:15.966 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:44:28.984 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:44:29.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:44:45.968 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:44:45.978 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:45:14.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:45:14.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:45:15.998 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:45:16.012 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:45:30.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:45:30.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:45:42.381 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE dox, scheduling delivery
2022-03-22 03:45:43.386 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox
2022-03-22 03:45:43.418 DEBUG 10 --- [uster-worker-23] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2022-03-22 03:45:43.432 DEBUG 10 --- [uster-worker-23] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 14 milliseconds
2022-03-22 03:45:47.014 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE zusammen_dox, scheduling delivery
2022-03-22 03:45:48.016 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox
2022-03-22 03:45:48.124 DEBUG 10 --- [uster-worker-24] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2022-03-22 03:45:48.154 DEBUG 10 --- [uster-worker-24] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 29 milliseconds
2022-03-22 03:45:55.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:45:55.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:45:56.932 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TYPE dox.version, scheduling delivery
2022-03-22 03:45:57.933 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.version (TYPE)
2022-03-22 03:46:00.520 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TYPE dox.user_candidate_version, scheduling delivery
2022-03-22 03:46:01.521 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.user_candidate_version (TYPE)
2022-03-22 03:46:02.024 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE workflow, scheduling delivery
2022-03-22 03:46:03.025 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for workflow
2022-03-22 03:46:03.107 DEBUG 10 --- [uster-worker-23] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2022-03-22 03:46:03.122 DEBUG 10 --- [uster-worker-23] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 14 milliseconds
2022-03-22 03:46:04.812 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE zusammen_workflow, scheduling delivery
2022-03-22 03:46:05.637 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.version_info, scheduling delivery
2022-03-22 03:46:06.638 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for everything.null (null)
2022-03-22 03:46:06.764 DEBUG 10 --- [uster-worker-24] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2022-03-22 03:46:06.782 DEBUG 10 --- [uster-worker-24] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 18 milliseconds
2022-03-22 03:46:11.426 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.version_info_deleted, scheduling delivery
2022-03-22 03:46:12.427 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.version_info_deleted (TABLE)
2022-03-22 03:46:12.716 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE workflow.unique_value, scheduling delivery
2022-03-22 03:46:13.716 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for workflow.unique_value (TABLE)
2022-03-22 03:46:23.515 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.unique_value, scheduling delivery
2022-03-22 03:46:24.515 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.unique_value (TABLE)
2022-03-22 03:46:27.924 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.item, scheduling delivery
2022-03-22 03:46:28.925 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_workflow.item (TABLE)
2022-03-22 03:46:32.104 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.package_details, scheduling delivery
2022-03-22 03:46:33.105 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.package_details (TABLE)
2022-03-22 03:46:34.728 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.version, scheduling delivery
2022-03-22 03:46:35.729 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_workflow.version (TABLE)
2022-03-22 03:46:36.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:46:36.627 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:46:40.213 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.version_elements, scheduling delivery
2022-03-22 03:46:41.214 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_workflow.version_elements (TABLE)
2022-03-22 03:46:41.224 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.vsp_enriched_service_template, scheduling delivery
2022-03-22 03:46:42.225 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.vsp_enriched_service_template (TABLE)
2022-03-22 03:46:52.209 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.vsp_enriched_service_artifact, scheduling delivery
2022-03-22 03:46:53.217 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.vsp_enriched_service_artifact (TABLE)
2022-03-22 03:46:56.719 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.element_namespace, scheduling delivery
2022-03-22 03:46:57.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:46:57.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:46:57.721 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_workflow.element_namespace (TABLE)
2022-03-22 03:47:00.317 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.application_config, scheduling delivery
2022-03-22 03:47:01.318 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.application_config (TABLE)
2022-03-22 03:47:02.512 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.element, scheduling delivery
2022-03-22 03:47:03.512 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_workflow.element (TABLE)
2022-03-22 03:47:05.713 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.element_synchronization_state, scheduling delivery
2022-03-22 03:47:06.353 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.action, scheduling delivery
2022-03-22 03:47:07.354 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for everything.null (null)
2022-03-22 03:47:07.424 DEBUG 10 --- [uster-worker-23] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2022-03-22 03:47:07.437 DEBUG 10 --- [uster-worker-23] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 13 milliseconds
2022-03-22 03:47:30.220 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2022-03-22 03:47:31.221 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2022-03-22 03:47:32.679 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:47:32.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:47:32.814 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.element_stage, scheduling delivery
2022-03-22 03:47:33.819 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_workflow.element_stage (TABLE)
2022-03-22 03:47:34.971 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2022-03-22 03:47:35.972 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2022-03-22 03:47:42.844 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.version_stage, scheduling delivery
2022-03-22 03:47:43.849 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_workflow.version_stage (TABLE)
2022-03-22 03:47:48.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:47:48.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:48:02.605 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2022-03-22 03:48:03.611 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2022-03-22 03:48:06.097 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2022-03-22 03:48:07.098 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2022-03-22 03:48:12.410 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2022-03-22 03:48:13.412 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2022-03-22 03:48:19.606 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.ecompcomponent, scheduling delivery
2022-03-22 03:48:20.607 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.ecompcomponent (TABLE)
2022-03-22 03:48:26.718 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2022-03-22 03:48:27.719 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2022-03-22 03:48:32.406 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.action_artifact, scheduling delivery
2022-03-22 03:48:33.407 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action_artifact (TABLE)
2022-03-22 03:48:37.076 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.activity_log, scheduling delivery
2022-03-22 03:48:38.077 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.activity_log (TABLE)
2022-03-22 03:48:40.837 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.healing, scheduling delivery
2022-03-22 03:48:41.838 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.healing (TABLE)
2022-03-22 03:48:43.861 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.migration, scheduling delivery
2022-03-22 03:48:44.862 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.migration (TABLE)
2022-03-22 03:48:47.705 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.item_permissions, scheduling delivery
2022-03-22 03:48:48.706 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.item_permissions (TABLE)
2022-03-22 03:48:55.312 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.user_permission_items, scheduling delivery
2022-03-22 03:48:56.313 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.user_permission_items (TABLE)
2022-03-22 03:48:58.819 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.notification_subscribers, scheduling delivery
2022-03-22 03:48:59.820 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.notification_subscribers (TABLE)
2022-03-22 03:49:02.015 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.last_notification, scheduling delivery
2022-03-22 03:49:03.021 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.last_notification (TABLE)
2022-03-22 03:49:08.917 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.notifications, scheduling delivery
2022-03-22 03:49:09.925 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.notifications (TABLE)
2022-03-22 03:49:19.318 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.vsp_merge_hint, scheduling delivery
2022-03-22 03:49:19.899 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:49:19.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:49:20.320 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.vsp_merge_hint (TABLE)
2022-03-22 03:49:29.120 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.item, scheduling delivery
2022-03-22 03:49:30.120 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.item (TABLE)
2022-03-22 03:49:30.600 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:49:30.672 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:49:35.045 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.version, scheduling delivery
2022-03-22 03:49:36.046 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.version (TABLE)
2022-03-22 03:49:39.195 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.version_elements, scheduling delivery
2022-03-22 03:49:40.195 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.version_elements (TABLE)
2022-03-22 03:50:02.715 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.element_namespace, scheduling delivery
2022-03-22 03:50:03.718 DEBUG 10 --- [uster-worker-24] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.element_namespace (TABLE)
2022-03-22 03:50:10.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:50:10.876 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:50:13.663 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.element, scheduling delivery
2022-03-22 03:50:14.679 DEBUG 10 --- [uster-worker-25] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.element (TABLE)
2022-03-22 03:50:36.007 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.element_synchronization_state, scheduling delivery
2022-03-22 03:50:37.013 DEBUG 10 --- [uster-worker-26] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.element_synchronization_state (TABLE)
2022-03-22 03:50:38.407 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.element_stage, scheduling delivery
2022-03-22 03:50:39.408 DEBUG 10 --- [uster-worker-25] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.element_stage (TABLE)
2022-03-22 03:50:47.710 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.version_stage, scheduling delivery
2022-03-22 03:50:48.711 DEBUG 10 --- [uster-worker-26] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.version_stage (TABLE)
2022-03-22 03:50:51.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:50:51.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:51:17.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:51:17.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:51:18.809 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:51:18.812 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:51:48.839 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:51:48.909 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:52:13.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:52:13.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:52:18.911 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:52:18.925 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:52:48.937 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:52:49.019 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:53:04.095 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:53:04.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:53:19.021 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:53:19.122 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:53:49.123 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:53:49.127 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:53:49.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:53:49.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:53:54.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:53:54.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:54:19.143 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:54:19.149 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:54:24.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:54:24.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:54:40.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:54:40.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:54:49.151 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:54:49.210 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:55:19.211 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:55:19.216 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:55:41.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:55:41.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:55:49.240 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:55:49.310 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:55:51.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:55:51.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:56:11.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:56:11.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:56:19.313 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:56:19.401 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:56:37.290 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:56:37.297 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:56:49.402 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:56:49.413 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:57:12.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:57:12.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:57:19.429 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:57:19.510 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:57:49.511 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:57:49.618 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:58:13.649 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:58:13.666 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:58:19.620 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:58:19.716 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:58:44.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:58:44.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:58:49.717 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:58:49.802 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:59:09.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:59:09.779 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:59:19.803 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:59:19.810 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:59:19.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:59:19.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:59:39.780 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:59:39.796 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 03:59:49.821 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 03:59:49.906 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:00:15.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:00:15.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:00:19.907 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:00:20.004 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:00:30.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:00:30.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:00:46.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:00:46.099 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:00:50.005 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:00:50.022 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:01:20.037 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:01:20.101 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:01:37.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:01:37.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:01:50.102 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:01:50.105 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:02:13.498 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:02:13.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:02:20.127 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:02:20.136 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:02:50.158 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:02:50.210 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:03:20.211 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:03:20.215 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:03:45.289 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:03:45.294 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:03:50.238 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:03:50.301 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:04:11.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:04:11.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:04:20.303 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:04:20.309 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:04:34.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:04:34.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] because: [/10.233.66.75:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2022-03-22 04:04:34.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preventing new connections for the next 1000 ms
2022-03-22 04:04:34.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-1, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:04:34.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-1, inFlight=0, closed=true] closing connection
2022-03-22 04:04:34.712 DEBUG 10 --- [uster-worker-27] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host DOWN
2022-03-22 04:04:34.714 DEBUG 10 --- [uster-worker-27] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 04:04:34.714 DEBUG 10 --- [uster-worker-27] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] starting reconnection attempt
2022-03-22 04:04:34.913 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.233.66.75:9042, scheduling delivery
2022-03-22 04:04:34.914 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event DOWN
2022-03-22 04:04:35.715 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:04:35.718 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-2, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:04:35.719 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-2, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:04:35.719 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-2, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:04:35.733 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-2, inFlight=0, closed=true] closing connection
2022-03-22 04:04:35.733 DEBUG 10 --- [uster-worker-28] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:04:35.734 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 2000 milliseconds
2022-03-22 04:04:35.734 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 2000 ms
2022-03-22 04:04:35.914 DEBUG 10 --- [uster-worker-27] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:04:37.734 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:04:37.737 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-3, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:04:37.738 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-3, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:04:37.738 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-3, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:04:37.738 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-3, inFlight=0, closed=true] closing connection
2022-03-22 04:04:37.739 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 4000 milliseconds
2022-03-22 04:04:37.739 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 4000 ms
2022-03-22 04:04:37.739 DEBUG 10 --- [uster-worker-28] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:04:41.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:04:41.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:04:41.743 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:04:41.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-4, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:04:41.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-4, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:04:41.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-4, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:04:41.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-4, inFlight=0, closed=true] closing connection
2022-03-22 04:04:41.748 DEBUG 10 --- [uster-worker-27] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:04:41.748 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 8000 milliseconds
2022-03-22 04:04:41.749 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 8000 ms
2022-03-22 04:04:49.749 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:04:49.752 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-5, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:04:49.753 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-5, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:04:49.753 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-5, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:04:49.753 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-5, inFlight=0, closed=true] closing connection
2022-03-22 04:04:49.753 DEBUG 10 --- [uster-worker-28] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:04:49.754 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 16000 milliseconds
2022-03-22 04:04:49.756 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 16000 ms
2022-03-22 04:05:04.922 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:05:04.925 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:05:05.756 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:05:05.771 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-6, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:05:05.771 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-6, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:05:05.772 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-6, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:05:05.772 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-6, inFlight=0, closed=true] closing connection
2022-03-22 04:05:05.773 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 04:05:05.773 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 32000 ms
2022-03-22 04:05:05.776 DEBUG 10 --- [uster-worker-27] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:05:34.949 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:05:34.952 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:05:37.774 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:05:37.800 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-7, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:05:37.801 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-7, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:05:37.801 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-7, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:05:37.810 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-7, inFlight=0, closed=true] closing connection
2022-03-22 04:05:37.810 DEBUG 10 --- [uster-worker-29] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:05:37.810 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 04:05:37.811 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 64000 ms
2022-03-22 04:06:04.972 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:06:05.006 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:06:20.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:06:20.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:06:35.007 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:06:35.010 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:06:41.811 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:06:41.863 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-8, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:06:41.864 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-8, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:06:41.864 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-8, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:06:41.867 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-8, inFlight=0, closed=true] closing connection
2022-03-22 04:06:41.868 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 04:06:41.868 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 128000 ms
2022-03-22 04:06:41.869 DEBUG 10 --- [uster-worker-30] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:07:05.025 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:07:05.105 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:07:35.106 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:07:35.116 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:08:05.120 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:08:05.202 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:08:35.204 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:08:35.224 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:08:49.869 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:08:49.885 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-9, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:08:49.885 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-9, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:08:49.886 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-9, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:08:49.886 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-9, inFlight=0, closed=true] closing connection
2022-03-22 04:08:49.887 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 256000 milliseconds
2022-03-22 04:08:49.887 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 256000 ms
2022-03-22 04:08:49.888 DEBUG 10 --- [uster-worker-31] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:09:05.229 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:09:05.305 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:09:35.306 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:09:35.318 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:10:05.336 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:10:05.402 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:10:25.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:10:25.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:10:35.403 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:10:35.408 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:10:51.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:10:51.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:11:05.421 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:11:05.424 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:11:35.454 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:11:35.479 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:12:05.481 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:12:05.504 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:12:35.506 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:12:35.510 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:13:05.511 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:13:05.517 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:13:05.888 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:13:05.898 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-10, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:13:05.899 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-10, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:13:05.899 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-10, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:13:05.901 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-10, inFlight=0, closed=true] closing connection
2022-03-22 04:13:05.903 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 512000 milliseconds
2022-03-22 04:13:05.903 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 512000 ms
2022-03-22 04:13:05.901 DEBUG 10 --- [uster-worker-32] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:13:14.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:13:14.379 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:13:35.541 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:13:35.554 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:14:05.571 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:14:05.574 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:14:35.600 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:14:35.611 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:15:05.618 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:15:05.723 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:15:09.819 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UP /10.233.66.75:9042, scheduling delivery
2022-03-22 04:15:09.820 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event UP
2022-03-22 04:15:10.830 DEBUG 10 --- [uster-worker-33] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.233.66.75:9042
2022-03-22 04:15:10.848 DEBUG 10 --- [uster-worker-33] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host UP
2022-03-22 04:15:10.848 DEBUG 10 --- [uster-worker-33] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2022-03-22 04:15:10.848 DEBUG 10 --- [uster-worker-33] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.233.66.75:9042
2022-03-22 04:15:10.848 DEBUG 10 --- [uster-worker-33] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:15:10.855 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-11, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:15:22.887 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-11, inFlight=0, closed=false] because: [/10.233.66.75:9042] Operation timed out
2022-03-22 04:15:22.888 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-11, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:15:22.888 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-11, inFlight=0, closed=true] closing connection
2022-03-22 04:15:22.889 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.66.75:9042-11, inFlight=0, closed=true]: there are still pending requests
2022-03-22 04:15:22.891 DEBUG 10 --- [uster-worker-33] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:15:22.895 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-12, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:15:37.815 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-12, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:15:37.819 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-12, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:15:37.819 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-12, inFlight=0, closed=true] closing connection
2022-03-22 04:15:37.822 WARN 10 --- [er-nio-worker-2] com.datastax.driver.core.Session : Error creating pool to /10.233.66.75:9042
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:15:37.825 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-12, inFlight=0, closed=true] has already terminated
2022-03-22 04:15:37.825 DEBUG 10 --- [uster-worker-33] com.datastax.driver.core.Cluster : Connection pool cannot be created, not marking /10.233.66.75:9042 UP
2022-03-22 04:15:37.826 DEBUG 10 --- [uster-worker-34] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host DOWN
2022-03-22 04:15:37.826 DEBUG 10 --- [uster-worker-34] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 04:15:37.826 DEBUG 10 --- [uster-worker-34] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] starting reconnection attempt
2022-03-22 04:15:37.826 DEBUG 10 --- [uster-worker-33] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:15:38.827 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:15:38.839 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-13, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:15:40.850 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:15:40.904 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:15:52.810 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-13, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:15:52.810 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-13, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:15:52.810 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-13, inFlight=0, closed=true] closing connection
2022-03-22 04:15:52.810 DEBUG 10 --- [uster-worker-34] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:15:52.811 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-13, inFlight=0, closed=true] has already terminated
2022-03-22 04:15:52.812 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:15:52.813 ERROR 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Authentication error during reconnection to /10.233.66.75:9042, scheduling retry in 2000 milliseconds
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:15:52.814 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 2000 ms
2022-03-22 04:15:53.302 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-11, inFlight=0, closed=true] has already terminated
2022-03-22 04:15:54.815 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:15:54.828 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-14, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:16:00.511 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-14, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:16:00.511 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-14, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:16:00.512 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-14, inFlight=0, closed=true] closing connection
2022-03-22 04:16:00.512 DEBUG 10 --- [uster-worker-33] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:16:00.513 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-14, inFlight=0, closed=true] has already terminated
2022-03-22 04:16:00.513 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:16:00.514 ERROR 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Authentication error during reconnection to /10.233.66.75:9042, scheduling retry in 4000 milliseconds
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:16:00.515 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 4000 ms
2022-03-22 04:16:04.517 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:16:04.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-15, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:16:10.905 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:16:10.910 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:16:11.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-15, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:16:11.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-15, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:16:11.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-15, inFlight=0, closed=true] closing connection
2022-03-22 04:16:11.110 DEBUG 10 --- [uster-worker-34] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:16:11.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-15, inFlight=0, closed=true] has already terminated
2022-03-22 04:16:11.111 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:16:11.111 ERROR 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Authentication error during reconnection to /10.233.66.75:9042, scheduling retry in 8000 milliseconds
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:16:11.113 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 8000 ms
2022-03-22 04:16:19.113 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:16:19.117 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-16, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:16:31.387 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-16, inFlight=0, closed=false] because: [/10.233.66.75:9042] Operation timed out
2022-03-22 04:16:31.387 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-16, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:16:31.388 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-16, inFlight=0, closed=true] closing connection
2022-03-22 04:16:31.388 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.66.75:9042-16, inFlight=0, closed=true]: there are still pending requests
2022-03-22 04:16:31.389 DEBUG 10 --- [uster-worker-35] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:16:31.389 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Operation timed out), scheduling retry in 16000 milliseconds
2022-03-22 04:16:31.389 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 16000 ms
2022-03-22 04:16:40.936 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:16:40.940 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:16:47.389 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:16:47.394 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-17, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:16:47.394 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-17, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:16:47.394 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-17, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:16:47.395 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-17, inFlight=0, closed=true] closing connection
2022-03-22 04:16:47.395 DEBUG 10 --- [uster-worker-36] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:16:47.396 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 04:16:47.396 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 32000 ms
2022-03-22 04:16:54.710 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.233.66.75:9042, scheduling delivery
2022-03-22 04:16:54.710 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event DOWN
2022-03-22 04:16:55.716 DEBUG 10 --- [uster-worker-35] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:17:08.303 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-16, inFlight=0, closed=true] has already terminated
2022-03-22 04:17:19.396 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:17:19.413 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-18, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:17:19.416 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-18, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:17:19.416 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-18, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:17:19.424 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-18, inFlight=0, closed=true] closing connection
2022-03-22 04:17:19.424 DEBUG 10 --- [uster-worker-37] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:17:19.425 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 04:17:19.425 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 64000 ms
2022-03-22 04:17:24.721 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:17:24.802 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:17:54.804 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:17:54.807 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:18:23.426 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:18:23.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-19, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:18:23.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-19, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:18:23.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-19, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:18:23.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-19, inFlight=0, closed=true] closing connection
2022-03-22 04:18:23.432 DEBUG 10 --- [uster-worker-38] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:18:23.433 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 04:18:23.433 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 128000 ms
2022-03-22 04:18:24.836 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:18:24.905 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:18:54.906 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:18:55.002 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:19:12.979 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:19:12.983 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:19:25.003 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:19:25.021 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:19:55.022 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:19:55.117 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:19:58.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:19:58.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:20:25.119 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:20:25.210 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:20:29.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:20:29.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:20:31.433 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:20:31.444 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-20, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:20:31.444 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-20, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:20:31.444 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-20, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:20:31.446 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-20, inFlight=0, closed=true] closing connection
2022-03-22 04:20:31.446 DEBUG 10 --- [uster-worker-39] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:20:31.446 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 256000 milliseconds
2022-03-22 04:20:31.446 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 256000 ms
2022-03-22 04:20:55.212 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:20:55.429 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:21:25.430 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:21:25.507 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:21:55.508 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:21:55.518 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:22:25.522 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:22:25.526 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:22:55.552 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:22:55.604 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:23:25.605 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:23:25.615 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:23:45.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:23:45.472 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:23:55.633 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:23:55.702 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:24:25.703 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:24:25.803 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:24:47.447 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:24:47.453 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-21, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:24:47.453 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-21, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:24:47.453 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-21, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:24:47.454 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-21, inFlight=0, closed=true] closing connection
2022-03-22 04:24:47.455 DEBUG 10 --- [uster-worker-40] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:24:47.459 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 512000 milliseconds
2022-03-22 04:24:47.459 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 512000 ms
2022-03-22 04:24:55.804 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:24:55.817 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:25:25.833 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:25:25.903 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:25:55.905 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:25:56.015 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:26:26.016 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:26:26.115 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:26:56.118 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:26:56.205 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:27:26.208 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:27:26.318 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:27:41.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:27:41.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:27:56.319 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:27:56.402 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:28:26.403 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:28:26.410 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:28:56.410 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:28:56.414 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:29:26.438 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:29:26.607 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:29:56.608 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:29:56.709 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:30:26.710 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:30:26.712 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:30:56.738 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:30:56.803 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:31:26.805 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:31:26.811 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:31:56.833 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:31:56.905 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:32:26.907 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:32:26.911 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:32:56.921 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:32:57.006 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:33:19.461 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:33:19.466 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-22, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:33:19.467 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-22, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:33:19.467 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-22, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:33:19.488 DEBUG 10 --- [uster-worker-41] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:33:19.490 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-22, inFlight=0, closed=true] closing connection
2022-03-22 04:33:19.492 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 04:33:19.492 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 600000 ms
2022-03-22 04:33:27.007 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:33:27.016 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:33:57.024 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:33:57.123 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:34:27.124 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:34:27.204 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:34:57.205 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:34:57.208 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:35:27.233 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:35:27.308 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:35:57.317 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:35:57.327 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:36:27.347 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:36:27.401 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:36:41.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:36:41.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:36:57.402 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:36:57.407 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:37:27.432 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:37:27.511 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:37:57.512 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:37:57.515 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:38:27.538 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:38:27.628 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:38:57.629 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:38:57.705 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:39:27.706 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:39:27.712 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:39:57.737 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:39:57.804 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:40:27.808 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:40:27.823 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:40:57.840 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:40:57.856 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:41:27.862 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:41:28.001 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:41:52.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:41:52.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:41:58.002 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:41:58.017 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:42:28.030 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:42:28.102 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:42:58.103 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:42:58.105 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:43:19.493 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:43:19.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-23, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:43:19.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-23, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:43:19.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-23, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:43:19.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-23, inFlight=0, closed=true] closing connection
2022-03-22 04:43:19.530 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 04:43:19.530 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 600000 ms
2022-03-22 04:43:19.531 DEBUG 10 --- [uster-worker-42] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:43:28.121 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:43:28.302 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:43:58.309 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:43:58.316 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:44:15.557 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:44:15.562 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:44:28.337 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:44:28.421 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:44:58.422 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:44:58.501 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:45:28.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:45:28.603 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:45:32.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:45:32.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:45:58.604 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:45:58.610 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:46:28.635 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:46:28.716 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:46:54.389 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:46:54.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:46:58.723 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:46:58.811 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:47:28.811 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:47:28.814 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:47:58.837 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:47:58.911 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:48:28.912 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:48:28.916 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:48:58.941 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:48:59.007 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:49:29.008 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:49:29.108 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:49:59.110 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:49:59.216 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:50:29.217 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:50:29.220 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:50:59.239 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:50:59.301 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:51:29.302 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:51:29.309 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:51:45.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:51:45.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:51:59.329 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:51:59.403 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:52:29.405 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:52:29.414 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:52:52.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:52:52.372 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:52:59.415 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:52:59.518 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:53:19.531 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:53:19.548 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-24, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:53:19.549 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-24, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:53:19.549 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-24, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:53:19.550 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-24, inFlight=0, closed=true] closing connection
2022-03-22 04:53:19.551 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 04:53:19.551 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 600000 ms
2022-03-22 04:53:19.551 DEBUG 10 --- [uster-worker-43] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:53:29.518 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:53:29.522 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:53:59.549 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:53:59.601 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:54:29.602 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:54:29.606 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:54:59.632 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:54:59.701 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:55:29.703 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:55:29.714 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:55:40.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:55:40.094 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:55:59.722 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:55:59.725 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:56:02.708 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UP /10.233.66.75:9042, scheduling delivery
2022-03-22 04:56:02.709 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event UP
2022-03-22 04:56:03.711 DEBUG 10 --- [uster-worker-44] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.233.66.75:9042
2022-03-22 04:56:03.728 DEBUG 10 --- [uster-worker-44] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host UP
2022-03-22 04:56:03.728 DEBUG 10 --- [uster-worker-44] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2022-03-22 04:56:03.728 DEBUG 10 --- [uster-worker-44] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.233.66.75:9042
2022-03-22 04:56:03.728 DEBUG 10 --- [uster-worker-44] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:56:03.733 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-25, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:56:20.488 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-25, inFlight=0, closed=false] because: [/10.233.66.75:9042] Operation timed out
2022-03-22 04:56:20.488 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-25, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:56:20.490 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-25, inFlight=0, closed=true] closing connection
2022-03-22 04:56:20.490 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.66.75:9042-25, inFlight=0, closed=true]: there are still pending requests
2022-03-22 04:56:20.491 DEBUG 10 --- [uster-worker-44] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:56:20.496 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-26, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:56:32.787 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-26, inFlight=0, closed=false] because: [/10.233.66.75:9042] Operation timed out
2022-03-22 04:56:32.787 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-26, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:56:32.788 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-26, inFlight=0, closed=true] closing connection
2022-03-22 04:56:32.788 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.66.75:9042-26, inFlight=0, closed=true]: there are still pending requests
2022-03-22 04:56:32.788 WARN 10 --- [ter-timeouter-0] c.d.driver.core.HostConnectionPool : Error creating connection to /10.233.66.75:9042
com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.233.66.75:9042] Operation timed out
at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1496)
at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1575)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:56:32.789 DEBUG 10 --- [ter-timeouter-0] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.233.66.75:9042 (1 connections needed, 0 successfully opened)
2022-03-22 04:56:32.789 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Session : Added connection pool for /10.233.66.75:9042
2022-03-22 04:56:32.790 DEBUG 10 --- [uster-worker-45] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host DOWN
2022-03-22 04:56:32.791 DEBUG 10 --- [uster-worker-45] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 04:56:32.791 DEBUG 10 --- [uster-worker-45] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] starting reconnection attempt
2022-03-22 04:56:32.791 DEBUG 10 --- [uster-worker-44] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:56:33.729 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:56:33.732 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:56:33.791 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:56:33.794 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-27, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:56:38.907 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-27, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:56:38.907 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-27, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:56:38.908 DEBUG 10 --- [uster-worker-45] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:56:38.908 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-27, inFlight=0, closed=true] closing connection
2022-03-22 04:56:38.909 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-27, inFlight=0, closed=true] has already terminated
2022-03-22 04:56:38.909 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:56:38.910 ERROR 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Authentication error during reconnection to /10.233.66.75:9042, scheduling retry in 2000 milliseconds
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:56:38.911 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 2000 ms
2022-03-22 04:56:40.912 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:56:40.914 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-28, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:56:46.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:56:46.035 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:56:47.307 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-28, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:56:47.307 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-28, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:56:47.307 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-28, inFlight=0, closed=true] closing connection
2022-03-22 04:56:47.307 DEBUG 10 --- [uster-worker-44] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:56:47.308 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-28, inFlight=0, closed=true] has already terminated
2022-03-22 04:56:47.308 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:56:47.311 ERROR 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Authentication error during reconnection to /10.233.66.75:9042, scheduling retry in 4000 milliseconds
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:56:47.313 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 4000 ms
2022-03-22 04:56:51.314 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:56:51.318 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-29, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:56:53.479 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-25, inFlight=0, closed=true] has already terminated
2022-03-22 04:56:56.707 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-29, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:56:56.707 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-29, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:56:56.708 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-29, inFlight=0, closed=true] closing connection
2022-03-22 04:56:56.708 DEBUG 10 --- [uster-worker-45] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:56:56.708 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-29, inFlight=0, closed=true] has already terminated
2022-03-22 04:56:56.708 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 04:56:56.709 ERROR 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Authentication error during reconnection to /10.233.66.75:9042, scheduling retry in 8000 milliseconds
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:56:56.710 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 8000 ms
2022-03-22 04:57:03.737 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:57:03.746 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:57:04.711 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:57:04.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-30, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 04:57:07.317 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.233.66.75:9042, scheduling delivery
2022-03-22 04:57:07.317 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event DOWN
2022-03-22 04:57:07.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-30, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 04:57:07.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-30, inFlight=0, closed=false] because: [/10.233.66.75:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2022-03-22 04:57:07.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-30, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:57:07.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-30, inFlight=0, closed=true] closing connection
2022-03-22 04:57:07.905 DEBUG 10 --- [uster-worker-44] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:57:07.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.66.75:9042-30, inFlight=0, closed=true]: there are still pending requests
2022-03-22 04:57:07.914 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Connection has been closed), scheduling retry in 16000 milliseconds
2022-03-22 04:57:07.914 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 16000 ms
2022-03-22 04:57:08.318 DEBUG 10 --- [uster-worker-45] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:57:08.480 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-26, inFlight=0, closed=true] has already terminated
2022-03-22 04:57:23.914 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:57:23.923 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-31, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:57:23.923 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-31, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:57:23.923 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-31, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:57:23.923 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-31, inFlight=0, closed=true] closing connection
2022-03-22 04:57:23.923 DEBUG 10 --- [uster-worker-44] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:57:23.942 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 04:57:23.942 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 32000 ms
2022-03-22 04:57:37.321 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:57:37.325 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:57:38.480 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-30, inFlight=0, closed=true] has already terminated
2022-03-22 04:57:55.943 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:57:55.954 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-32, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:57:55.955 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-32, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:57:55.955 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-32, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:57:55.956 DEBUG 10 --- [uster-worker-46] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:57:55.956 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-32, inFlight=0, closed=true] closing connection
2022-03-22 04:57:55.957 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 04:57:55.957 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 64000 ms
2022-03-22 04:58:07.333 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:58:07.417 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:58:37.418 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:58:37.448 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:58:59.958 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 04:58:59.986 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-33, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 04:58:59.986 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-33, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 04:58:59.986 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-33, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 04:58:59.988 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-33, inFlight=0, closed=true] closing connection
2022-03-22 04:58:59.989 DEBUG 10 --- [uster-worker-47] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 04:58:59.997 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 04:58:59.998 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 128000 ms
2022-03-22 04:59:07.449 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:59:07.458 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 04:59:37.472 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 04:59:37.486 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:00:07.487 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:00:07.494 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:00:37.516 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:00:37.532 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:01:07.535 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:01:07.538 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:01:07.998 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:01:08.041 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-34, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:01:08.041 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-34, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:01:08.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-34, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:01:08.073 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-34, inFlight=0, closed=true] closing connection
2022-03-22 05:01:08.074 DEBUG 10 --- [uster-worker-48] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:01:08.085 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 256000 milliseconds
2022-03-22 05:01:08.085 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 256000 ms
2022-03-22 05:01:37.565 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:01:37.568 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:01:52.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:01:52.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:01:56.201 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UP /10.233.66.75:9042, scheduling delivery
2022-03-22 05:01:56.202 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event UP
2022-03-22 05:01:57.208 DEBUG 10 --- [uster-worker-49] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.233.66.75:9042
2022-03-22 05:01:57.213 DEBUG 10 --- [uster-worker-49] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host UP
2022-03-22 05:01:57.214 DEBUG 10 --- [uster-worker-49] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2022-03-22 05:01:57.214 DEBUG 10 --- [uster-worker-49] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.233.66.75:9042
2022-03-22 05:01:57.214 DEBUG 10 --- [uster-worker-49] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:01:57.217 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-35, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:02:09.406 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-35, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 05:02:09.406 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-35, inFlight=0, closed=false] because: [/10.233.66.75:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2022-03-22 05:02:09.406 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-35, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:02:09.407 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-35, inFlight=0, closed=true] closing connection
2022-03-22 05:02:09.408 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.66.75:9042-35, inFlight=0, closed=true]: there are still pending requests
2022-03-22 05:02:09.409 DEBUG 10 --- [uster-worker-49] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:02:09.416 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-36, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:02:09.416 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-36, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:02:09.416 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-36, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:02:09.416 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-36, inFlight=0, closed=true] closing connection
2022-03-22 05:02:09.417 WARN 10 --- [er-nio-worker-3] c.d.driver.core.HostConnectionPool : Error creating connection to /10.233.66.75:9042
com.datastax.driver.core.exceptions.TransportException: [/10.233.66.75:9042] Cannot connect
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:224)
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:194)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.66.75:9042
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 05:02:09.418 DEBUG 10 --- [er-nio-worker-3] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.233.66.75:9042 (1 connections needed, 0 successfully opened)
2022-03-22 05:02:09.418 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Session : Added connection pool for /10.233.66.75:9042
2022-03-22 05:02:09.419 DEBUG 10 --- [uster-worker-50] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host DOWN
2022-03-22 05:02:09.420 DEBUG 10 --- [uster-worker-50] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 05:02:09.420 DEBUG 10 --- [uster-worker-50] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] starting reconnection attempt
2022-03-22 05:02:09.420 DEBUG 10 --- [uster-worker-49] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:02:10.105 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.233.66.75:9042, scheduling delivery
2022-03-22 05:02:10.106 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event DOWN
2022-03-22 05:02:10.420 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:02:10.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-37, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:02:10.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-37, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:02:10.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-37, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:02:10.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-37, inFlight=0, closed=true] closing connection
2022-03-22 05:02:10.425 DEBUG 10 --- [uster-worker-49] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:02:10.426 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 2000 milliseconds
2022-03-22 05:02:10.426 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 2000 ms
2022-03-22 05:02:11.107 DEBUG 10 --- [uster-worker-50] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:02:12.427 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:02:12.431 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-38, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:02:12.431 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-38, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:02:12.431 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-38, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:02:12.431 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-38, inFlight=0, closed=true] closing connection
2022-03-22 05:02:12.431 DEBUG 10 --- [uster-worker-49] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:02:12.433 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 4000 milliseconds
2022-03-22 05:02:12.433 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 4000 ms
2022-03-22 05:02:16.433 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:02:16.437 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-39, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:02:16.437 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-39, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:02:16.437 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-39, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:02:16.437 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-39, inFlight=0, closed=true] closing connection
2022-03-22 05:02:16.438 DEBUG 10 --- [uster-worker-50] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:02:16.439 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 8000 milliseconds
2022-03-22 05:02:16.439 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 8000 ms
2022-03-22 05:02:24.439 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:02:24.444 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-40, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:02:24.444 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-40, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:02:24.444 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-40, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:02:24.444 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-40, inFlight=0, closed=true] closing connection
2022-03-22 05:02:24.445 DEBUG 10 --- [uster-worker-49] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:02:24.445 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 16000 milliseconds
2022-03-22 05:02:24.445 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 16000 ms
2022-03-22 05:02:38.487 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-35, inFlight=0, closed=true] has already terminated
2022-03-22 05:02:40.117 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:02:40.120 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:02:40.446 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:02:40.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-41, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:02:40.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-41, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:02:40.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-41, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:02:40.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-41, inFlight=0, closed=true] closing connection
2022-03-22 05:02:40.450 DEBUG 10 --- [uster-worker-50] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:02:40.450 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 05:02:40.451 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 32000 ms
2022-03-22 05:03:10.122 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:03:10.129 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:03:12.451 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:03:12.485 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-42, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:03:12.486 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-42, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:03:12.486 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-42, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:03:12.488 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-42, inFlight=0, closed=true] closing connection
2022-03-22 05:03:12.488 DEBUG 10 --- [uster-worker-51] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:03:12.488 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 05:03:12.488 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 64000 ms
2022-03-22 05:03:40.149 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:03:40.164 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:03:44.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:03:44.743 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:04:10.173 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:04:10.177 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:04:16.489 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:04:16.505 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-43, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:04:16.505 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-43, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:04:16.505 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-43, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:04:16.506 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-43, inFlight=0, closed=true] closing connection
2022-03-22 05:04:16.506 DEBUG 10 --- [uster-worker-52] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:04:16.507 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 05:04:16.507 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 128000 ms
2022-03-22 05:04:40.197 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:04:40.212 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:05:10.215 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:05:10.219 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:05:40.237 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:05:40.254 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:05:42.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:05:42.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:06:10.259 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:06:10.271 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:06:24.508 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:06:24.523 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-44, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:06:24.524 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-44, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:06:24.524 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-44, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:06:24.525 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-44, inFlight=0, closed=true] closing connection
2022-03-22 05:06:24.526 DEBUG 10 --- [uster-worker-53] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:06:24.528 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 256000 milliseconds
2022-03-22 05:06:24.529 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 256000 ms
2022-03-22 05:06:28.599 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:06:28.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:06:40.272 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:06:40.278 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:07:10.295 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:07:10.302 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:07:40.325 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:07:40.329 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:08:10.355 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:08:10.359 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:08:40.380 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:08:40.385 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:09:10.409 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:09:10.422 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:09:40.432 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:09:40.457 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:10:10.493 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:10:10.511 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:10:40.521 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:10:40.530 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:10:40.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-45, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:10:40.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-45, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:10:40.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-45, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:10:40.544 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-45, inFlight=0, closed=true] closing connection
2022-03-22 05:10:40.544 DEBUG 10 --- [uster-worker-54] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:10:40.545 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 512000 milliseconds
2022-03-22 05:10:40.545 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 512000 ms
2022-03-22 05:10:40.549 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:11:10.549 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:11:10.581 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:11:40.582 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:11:40.585 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:12:10.620 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:12:10.713 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:12:39.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:12:39.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:12:40.714 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:12:41.105 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:12:52.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 05:12:52.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] because: [/10.233.71.64:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2022-03-22 05:12:52.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preventing new connections for the next 1000 ms
2022-03-22 05:12:52.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-23, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:12:52.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-23, inFlight=0, closed=true] closing connection
2022-03-22 05:12:52.714 DEBUG 10 --- [uster-worker-55] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host DOWN
2022-03-22 05:12:52.715 DEBUG 10 --- [uster-worker-55] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 05:12:52.715 DEBUG 10 --- [uster-worker-55] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] starting reconnection attempt
2022-03-22 05:12:52.931 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.233.71.64:9042, scheduling delivery
2022-03-22 05:12:52.931 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] received event DOWN
2022-03-22 05:12:53.717 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:12:53.747 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-24, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:12:53.748 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-24, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:12:53.748 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-24, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:12:53.748 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-24, inFlight=0, closed=true] closing connection
2022-03-22 05:12:53.749 DEBUG 10 --- [uster-worker-56] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:12:53.749 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 2000 milliseconds
2022-03-22 05:12:53.750 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 2000 ms
2022-03-22 05:12:53.932 DEBUG 10 --- [uster-worker-55] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:12:55.751 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:12:55.774 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-25, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:12:55.775 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-25, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:12:55.775 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-25, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:12:55.775 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-25, inFlight=0, closed=true] closing connection
2022-03-22 05:12:55.777 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 4000 milliseconds
2022-03-22 05:12:55.777 DEBUG 10 --- [uster-worker-56] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:12:55.777 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 4000 ms
2022-03-22 05:12:59.782 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:12:59.792 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-26, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:12:59.793 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-26, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:12:59.793 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-26, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:12:59.793 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-26, inFlight=0, closed=true] closing connection
2022-03-22 05:12:59.794 DEBUG 10 --- [uster-worker-55] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:12:59.794 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 8000 milliseconds
2022-03-22 05:12:59.794 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 8000 ms
2022-03-22 05:13:01.492 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:12:50.722Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:13:07.795 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:13:07.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-27, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:13:07.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-27, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:13:07.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-27, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:13:07.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-27, inFlight=0, closed=true] closing connection
2022-03-22 05:13:07.803 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 16000 milliseconds
2022-03-22 05:13:07.803 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 16000 ms
2022-03-22 05:13:07.803 DEBUG 10 --- [uster-worker-56] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:13:11.525 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:12:50.722Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:13:21.618 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:12:50.722Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:13:22.941 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:13:23.015 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:13:23.803 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:13:23.812 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-28, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:13:23.812 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-28, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:13:23.813 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-28, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:13:23.813 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-28, inFlight=0, closed=true] closing connection
2022-03-22 05:13:23.813 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 05:13:23.813 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 32000 ms
2022-03-22 05:13:23.813 DEBUG 10 --- [uster-worker-55] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:13:31.744 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:12:50.722Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:13:41.783 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:12:50.722Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:13:51.816 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:12:50.722Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:13:53.017 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:13:53.110 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:13:55.814 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:13:55.831 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-29, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:13:55.831 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-29, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:13:55.831 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-29, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:13:55.832 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-29, inFlight=0, closed=true] closing connection
2022-03-22 05:13:55.832 DEBUG 10 --- [uster-worker-57] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:13:55.849 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 05:13:55.849 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 64000 ms
2022-03-22 05:14:01.912 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:12:50.722Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:14:11.938 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:12:50.722Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:14:23.112 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:14:23.204 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:14:31.507 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:14:41.527 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:14:51.610 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:14:53.205 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:14:53.216 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:14:59.849 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:14:59.865 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-30, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:14:59.866 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-30, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:14:59.866 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-30, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:14:59.867 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-30, inFlight=0, closed=true] closing connection
2022-03-22 05:14:59.868 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 05:14:59.868 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 128000 ms
2022-03-22 05:14:59.868 DEBUG 10 --- [uster-worker-58] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:15:01.820 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:15:11.920 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:15:22.020 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:15:23.227 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:15:23.318 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:15:32.113 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:15:32.509 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UP /10.233.66.75:9042, scheduling delivery
2022-03-22 05:15:32.510 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event UP
2022-03-22 05:15:33.513 DEBUG 10 --- [uster-worker-59] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.233.66.75:9042
2022-03-22 05:15:33.522 DEBUG 10 --- [uster-worker-59] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host UP
2022-03-22 05:15:33.523 DEBUG 10 --- [uster-worker-59] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2022-03-22 05:15:33.523 DEBUG 10 --- [uster-worker-59] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.233.66.75:9042
2022-03-22 05:15:33.523 DEBUG 10 --- [uster-worker-59] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:15:33.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-46, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:15:42.124 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:15:45.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-46, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 05:15:45.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-46, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:15:45.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-46, inFlight=0, closed=true] closing connection
2022-03-22 05:15:45.755 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-46, inFlight=0, closed=true] has already terminated
2022-03-22 05:15:45.757 DEBUG 10 --- [uster-worker-59] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:15:45.781 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-47, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:15:51.312 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-47, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 05:15:51.312 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-47, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:15:51.313 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-47, inFlight=0, closed=true] closing connection
2022-03-22 05:15:51.314 WARN 10 --- [er-nio-worker-0] com.datastax.driver.core.Session : Error creating pool to /10.233.66.75:9042
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 05:15:51.315 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-47, inFlight=0, closed=true] has already terminated
2022-03-22 05:15:51.318 DEBUG 10 --- [uster-worker-59] com.datastax.driver.core.Cluster : Connection pool cannot be created, not marking /10.233.66.75:9042 UP
2022-03-22 05:15:51.318 DEBUG 10 --- [uster-worker-60] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host DOWN
2022-03-22 05:15:51.319 DEBUG 10 --- [uster-worker-60] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 05:15:51.319 DEBUG 10 --- [uster-worker-60] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] starting reconnection attempt
2022-03-22 05:15:51.319 DEBUG 10 --- [uster-worker-59] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:15:52.205 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:15:52.319 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:15:52.335 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-48, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:15:58.199 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-48, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 05:15:58.199 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-48, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:15:58.199 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-48, inFlight=0, closed=true] closing connection
2022-03-22 05:15:58.199 DEBUG 10 --- [uster-worker-59] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:15:58.200 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-48, inFlight=0, closed=true] has already terminated
2022-03-22 05:15:58.200 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 05:15:58.200 ERROR 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Authentication error during reconnection to /10.233.66.75:9042, scheduling retry in 2000 milliseconds
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 05:15:58.202 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 2000 ms
2022-03-22 05:16:00.204 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:16:00.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-49, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:16:02.217 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:13:20Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:16:03.532 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:16:03.538 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:16:13.888 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-49, inFlight=0, closed=false] because: [/10.233.66.75:9042] Operation timed out
2022-03-22 05:16:13.888 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-49, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:16:13.888 DEBUG 10 --- [uster-worker-60] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:16:13.888 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-49, inFlight=0, closed=true] closing connection
2022-03-22 05:16:13.888 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.66.75:9042-49, inFlight=0, closed=true]: there are still pending requests
2022-03-22 05:16:13.889 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Operation timed out), scheduling retry in 4000 milliseconds
2022-03-22 05:16:13.889 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 4000 ms
2022-03-22 05:16:17.889 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:16:17.911 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-50, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:16:21.134 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:16:29.987 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-50, inFlight=0, closed=false] because: [/10.233.66.75:9042] Operation timed out
2022-03-22 05:16:29.987 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-50, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:16:29.988 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-50, inFlight=0, closed=true] closing connection
2022-03-22 05:16:29.988 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.66.75:9042-50, inFlight=0, closed=true]: there are still pending requests
2022-03-22 05:16:29.988 DEBUG 10 --- [uster-worker-61] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:16:29.989 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Operation timed out), scheduling retry in 8000 milliseconds
2022-03-22 05:16:29.989 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 8000 ms
2022-03-22 05:16:31.205 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:16:33.538 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:16:33.543 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:16:37.989 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:16:38.010 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-51, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:16:38.010 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-51, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:16:38.010 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-51, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:16:38.011 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-51, inFlight=0, closed=true] closing connection
2022-03-22 05:16:38.012 DEBUG 10 --- [uster-worker-60] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:16:38.013 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 16000 milliseconds
2022-03-22 05:16:38.013 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 16000 ms
2022-03-22 05:16:38.566 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-49, inFlight=0, closed=true] has already terminated
2022-03-22 05:16:41.221 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:16:42.846 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.233.66.75:9042, scheduling delivery
2022-03-22 05:16:42.846 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event DOWN
2022-03-22 05:16:43.847 DEBUG 10 --- [uster-worker-61] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:16:46.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836184123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e3482dd at consistency null
2022-03-22 05:16:46.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836184123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:46.285 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171675563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ede84cc at consistency null
2022-03-22 05:16:46.285 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171675563-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:46.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [976407956-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65d0b2e6 at consistency null
2022-03-22 05:16:46.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [976407956-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:46.378 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1114988799-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4727905c at consistency null
2022-03-22 05:16:46.378 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1114988799-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:46.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [544715424-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d8fc00f at consistency null
2022-03-22 05:16:46.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [544715424-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:46.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1986471890-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47ba1411 at consistency null
2022-03-22 05:16:46.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1986471890-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:46.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1511332583-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ce4d9ca at consistency null
2022-03-22 05:16:46.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1511332583-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:46.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [856619970-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f58e9e1 at consistency null
2022-03-22 05:16:46.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [856619970-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:46.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779201969-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46203cf9 at consistency null
2022-03-22 05:16:46.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779201969-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:47.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1634162202-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@512da2b0 at consistency null
2022-03-22 05:16:47.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1634162202-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:47.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [324512499-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b1d1bd6 at consistency null
2022-03-22 05:16:47.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [324512499-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:47.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1472435120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c2f4b5 at consistency null
2022-03-22 05:16:47.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1472435120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:48.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1962713308-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7846c5ca at consistency null
2022-03-22 05:16:48.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1962713308-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:48.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804323295-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ba6a2bf at consistency null
2022-03-22 05:16:48.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804323295-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:49.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [22993120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dfba0df at consistency null
2022-03-22 05:16:49.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [22993120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:49.715 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:16:54.014 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:16:54.024 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-52, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:16:54.026 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-52, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:16:54.026 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-52, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:16:54.026 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-52, inFlight=0, closed=true] closing connection
2022-03-22 05:16:54.027 DEBUG 10 --- [uster-worker-60] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:16:54.027 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 05:16:54.027 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 32000 ms
2022-03-22 05:16:54.834 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1246467432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53dd4e3f at consistency null
2022-03-22 05:16:54.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1246467432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:54.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1528833725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d00bda1 at consistency null
2022-03-22 05:16:54.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1528833725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:55.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1929786369-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ea76ca1 at consistency null
2022-03-22 05:16:55.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1929786369-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:55.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [73952089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e1ac2f2 at consistency null
2022-03-22 05:16:55.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [73952089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:55.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [312368745-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@278fdbb8 at consistency null
2022-03-22 05:16:55.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [312368745-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:55.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213302601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fda875d at consistency null
2022-03-22 05:16:55.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213302601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:56.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171246718-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a7d3263 at consistency null
2022-03-22 05:16:56.132 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171246718-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:56.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1296078018-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32b8728c at consistency null
2022-03-22 05:16:56.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1296078018-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:57.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768468501-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bb33d78 at consistency null
2022-03-22 05:16:57.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768468501-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:57.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776441130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2018bf3c at consistency null
2022-03-22 05:16:57.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776441130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:58.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1263970460-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33e57f76 at consistency null
2022-03-22 05:16:58.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1263970460-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:16:58.715 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:17:03.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49922992-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42384794 at consistency null
2022-03-22 05:17:03.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49922992-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:03.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1086746804-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3092fa56 at consistency null
2022-03-22 05:17:03.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1086746804-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:04.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [996553857-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55964227 at consistency null
2022-03-22 05:17:04.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [996553857-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:04.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1166698809-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d90892c at consistency null
2022-03-22 05:17:04.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1166698809-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:04.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [161666683-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d9473ed at consistency null
2022-03-22 05:17:04.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [161666683-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:04.382 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2048017451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a736043 at consistency null
2022-03-22 05:17:04.382 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2048017451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:04.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893377492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d939050 at consistency null
2022-03-22 05:17:04.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893377492-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:04.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1720703319-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4403a127 at consistency null
2022-03-22 05:17:04.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1720703319-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:05.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038549235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a66b951 at consistency null
2022-03-22 05:17:05.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038549235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:05.294 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299559668-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5be6b526 at consistency null
2022-03-22 05:17:05.294 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299559668-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:05.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873142558-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a66af1a at consistency null
2022-03-22 05:17:05.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873142558-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:06.157 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743848873-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@133535e8 at consistency null
2022-03-22 05:17:06.157 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743848873-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:06.966 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1542961794-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c86aaee at consistency null
2022-03-22 05:17:06.966 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1542961794-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:06.967 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:17:07.868 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:17:07.890 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-31, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:17:07.891 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-31, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:17:07.891 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-31, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:17:07.891 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-31, inFlight=0, closed=true] closing connection
2022-03-22 05:17:07.891 DEBUG 10 --- [uster-worker-61] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:17:07.892 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 256000 milliseconds
2022-03-22 05:17:07.892 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 256000 ms
2022-03-22 05:17:08.567 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-50, inFlight=0, closed=true] has already terminated
2022-03-22 05:17:11.988 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [498890306-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2766e0c9 at consistency null
2022-03-22 05:17:11.988 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [498890306-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1897090818-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e8eef11 at consistency null
2022-03-22 05:17:12.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1897090818-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.075 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1812310522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@477e4c94 at consistency null
2022-03-22 05:17:12.075 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1812310522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180366674-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a72097b at consistency null
2022-03-22 05:17:12.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180366674-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957712942-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a02cc54 at consistency null
2022-03-22 05:17:12.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957712942-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [531660517-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@139d5262 at consistency null
2022-03-22 05:17:12.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [531660517-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.293 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1589337744-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26b2a669 at consistency null
2022-03-22 05:17:12.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1589337744-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436177064-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c714a15 at consistency null
2022-03-22 05:17:12.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436177064-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.526 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603811089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@307d5da3 at consistency null
2022-03-22 05:17:12.526 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603811089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067874525-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@498a078c at consistency null
2022-03-22 05:17:12.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067874525-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:12.855 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:17:12.860 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:17:12.954 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005448807-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ab8613a at consistency null
2022-03-22 05:17:12.954 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005448807-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:13.359 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013314720-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ccfbd9a at consistency null
2022-03-22 05:17:13.359 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013314720-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:13.781 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1949864920-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76057de5 at consistency null
2022-03-22 05:17:13.781 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1949864920-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:14.187 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2018798866-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6faab2ce at consistency null
2022-03-22 05:17:14.188 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2018798866-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:14.594 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [748563113-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5622afb0 at consistency null
2022-03-22 05:17:14.594 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [748563113-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:15.400 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070406510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d0fb83 at consistency null
2022-03-22 05:17:15.400 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070406510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:15.403 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:17:20.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266074396-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ae744c9 at consistency null
2022-03-22 05:17:20.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266074396-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:20.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1413232904-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6061a199 at consistency null
2022-03-22 05:17:20.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1413232904-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:20.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [153782852-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23fddd0a at consistency null
2022-03-22 05:17:20.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [153782852-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:20.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1592522340-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23712615 at consistency null
2022-03-22 05:17:20.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1592522340-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:20.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [237561788-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4350efbe at consistency null
2022-03-22 05:17:20.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [237561788-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:21.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005153032-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e50f324 at consistency null
2022-03-22 05:17:21.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005153032-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:21.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1724694171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63087e69 at consistency null
2022-03-22 05:17:21.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1724694171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:21.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1729445104-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49e3ad93 at consistency null
2022-03-22 05:17:21.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1729445104-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:21.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776549033-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d8a960e at consistency null
2022-03-22 05:17:21.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776549033-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:22.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1586019651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@246a8d48 at consistency null
2022-03-22 05:17:22.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1586019651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:23.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572075631-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@431449a at consistency null
2022-03-22 05:17:23.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572075631-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:23.109 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:17:26.028 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:17:26.040 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-53, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:17:26.040 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-53, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:17:26.040 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-53, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:17:26.042 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-53, inFlight=0, closed=true] closing connection
2022-03-22 05:17:26.043 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 05:17:26.043 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 64000 ms
2022-03-22 05:17:26.043 DEBUG 10 --- [uster-worker-62] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:17:28.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [319794581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@989284f at consistency null
2022-03-22 05:17:28.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [319794581-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:28.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1949571367-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79f9f4fb at consistency null
2022-03-22 05:17:28.438 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1949571367-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:28.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1875759175-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77e26aca at consistency null
2022-03-22 05:17:28.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1875759175-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:28.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [771702097-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70fad5b2 at consistency null
2022-03-22 05:17:28.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [771702097-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:28.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1131564670-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30f140b2 at consistency null
2022-03-22 05:17:28.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1131564670-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:28.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391879190-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@699247b at consistency null
2022-03-22 05:17:28.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391879190-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:29.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [378875730-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62c9ba58 at consistency null
2022-03-22 05:17:29.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [378875730-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:29.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1121226084-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4438a4f6 at consistency null
2022-03-22 05:17:29.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1121226084-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:29.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [319044350-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ef69813 at consistency null
2022-03-22 05:17:29.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [319044350-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:29.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1489275298-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3103c14a at consistency null
2022-03-22 05:17:29.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1489275298-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:30.132 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1737802327-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e37b648 at consistency null
2022-03-22 05:17:30.132 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1737802327-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:31.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [30268524-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f29ae6c at consistency null
2022-03-22 05:17:31.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [30268524-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:31.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866964833-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55558b9a at consistency null
2022-03-22 05:17:31.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866964833-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:31.852 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:17:36.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1339671343-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7af309eb at consistency null
2022-03-22 05:17:36.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1339671343-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:37.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245245689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5955750b at consistency null
2022-03-22 05:17:37.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245245689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:37.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1633662458-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bddea83 at consistency null
2022-03-22 05:17:37.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1633662458-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:37.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873296749-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72ada939 at consistency null
2022-03-22 05:17:37.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873296749-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:37.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [902569592-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10f35ad6 at consistency null
2022-03-22 05:17:37.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [902569592-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:38.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1251626966-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62fc88ee at consistency null
2022-03-22 05:17:38.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1251626966-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:38.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1450627866-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@611efc0 at consistency null
2022-03-22 05:17:38.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1450627866-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:38.655 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155145378-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@196a7a5a at consistency null
2022-03-22 05:17:38.655 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155145378-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:39.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1557505059-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13c5b64b at consistency null
2022-03-22 05:17:39.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1557505059-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:39.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151972496-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1578e725 at consistency null
2022-03-22 05:17:39.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151972496-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:39.919 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:17:42.885 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:17:42.903 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:17:45.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [614940956-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c1e75a5 at consistency null
2022-03-22 05:17:45.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [614940956-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:45.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1464769390-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56ec74d6 at consistency null
2022-03-22 05:17:45.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1464769390-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:45.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [642504984-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6825f7ae at consistency null
2022-03-22 05:17:45.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [642504984-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:45.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [802157685-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a367440 at consistency null
2022-03-22 05:17:45.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [802157685-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:45.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [830406449-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55ea9dfe at consistency null
2022-03-22 05:17:45.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [830406449-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:45.553 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315131803-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60151951 at consistency null
2022-03-22 05:17:45.553 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315131803-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:45.762 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1790123971-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27ca1a7a at consistency null
2022-03-22 05:17:45.763 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1790123971-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:46.183 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [982463164-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ed7eeb7 at consistency null
2022-03-22 05:17:46.183 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [982463164-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:46.647 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2093929899-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fd383fe at consistency null
2022-03-22 05:17:46.647 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2093929899-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:47.463 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [711750134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6744c8c4 at consistency null
2022-03-22 05:17:47.463 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [711750134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:48.298 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1481850495-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10aa179c at consistency null
2022-03-22 05:17:48.299 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1481850495-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:48.300 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:17:53.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [462093595-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@199f168 at consistency null
2022-03-22 05:17:53.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [462093595-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:53.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [300089420-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32e64ed6 at consistency null
2022-03-22 05:17:53.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [300089420-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:53.526 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1481191495-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73a5b2a6 at consistency null
2022-03-22 05:17:53.526 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1481191495-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:53.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [422071799-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33734db9 at consistency null
2022-03-22 05:17:53.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [422071799-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:53.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1434847724-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dafe80e at consistency null
2022-03-22 05:17:53.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1434847724-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:54.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1559138928-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dbf9993 at consistency null
2022-03-22 05:17:54.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1559138928-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:54.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1303665314-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23512400 at consistency null
2022-03-22 05:17:54.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1303665314-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:55.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [380170518-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ae6ab56 at consistency null
2022-03-22 05:17:55.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [380170518-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:56.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1938047814-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@88bdd9c at consistency null
2022-03-22 05:17:56.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1938047814-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:57.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1540432468-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@979be1f at consistency null
2022-03-22 05:17:57.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1540432468-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:17:57.021 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:18:02.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [887030470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2322ea14 at consistency null
2022-03-22 05:18:02.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [887030470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:02.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1697539338-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ca1493d at consistency null
2022-03-22 05:18:02.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1697539338-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:02.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [887198874-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a101008 at consistency null
2022-03-22 05:18:02.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [887198874-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:02.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727054800-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e5dc613 at consistency null
2022-03-22 05:18:02.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727054800-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:02.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720712954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a1bff86 at consistency null
2022-03-22 05:18:02.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720712954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:02.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013256859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ce5b159 at consistency null
2022-03-22 05:18:02.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013256859-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:03.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [77075657-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@663f749d at consistency null
2022-03-22 05:18:03.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [77075657-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:03.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1687595465-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b24b17 at consistency null
2022-03-22 05:18:03.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1687595465-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:03.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741099845-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63b07388 at consistency null
2022-03-22 05:18:03.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741099845-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:04.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931617516-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62de69de at consistency null
2022-03-22 05:18:04.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931617516-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:04.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [928532509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fd86284 at consistency null
2022-03-22 05:18:04.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [928532509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:04.918 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:18:09.967 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1716615134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b26649b at consistency null
2022-03-22 05:18:09.967 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1716615134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:10.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961744234-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bea6644 at consistency null
2022-03-22 05:18:10.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961744234-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:10.142 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1530158056-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1aeb6e7e at consistency null
2022-03-22 05:18:10.142 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1530158056-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:10.253 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [786036787-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@261bf3e5 at consistency null
2022-03-22 05:18:10.254 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [786036787-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:10.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1130075569-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ec15901 at consistency null
2022-03-22 05:18:10.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1130075569-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:10.596 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1648556269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2641c8db at consistency null
2022-03-22 05:18:10.596 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1648556269-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:11.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466617430-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6718caf3 at consistency null
2022-03-22 05:18:11.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466617430-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:11.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049539967-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ee5892e at consistency null
2022-03-22 05:18:11.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049539967-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:12.914 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:18:12.929 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:18:13.453 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1698728854-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a98c2a2 at consistency null
2022-03-22 05:18:13.453 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1698728854-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:13.454 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:18:18.473 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [280007235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42b7c83a at consistency null
2022-03-22 05:18:18.473 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [280007235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:18.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [198825655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51c78461 at consistency null
2022-03-22 05:18:18.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [198825655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:18.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985305652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d66d129 at consistency null
2022-03-22 05:18:18.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985305652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:18.939 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1539814094-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ec30a7 at consistency null
2022-03-22 05:18:18.939 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1539814094-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:19.365 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1332810111-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c7622f2 at consistency null
2022-03-22 05:18:19.365 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1332810111-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:19.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [173541960-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41ff4f91 at consistency null
2022-03-22 05:18:19.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [173541960-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:20.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1688011269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a5b311e at consistency null
2022-03-22 05:18:20.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1688011269-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:21.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [44735898-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a5251fa at consistency null
2022-03-22 05:18:21.544 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [44735898-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:21.548 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:18:26.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1159171673-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77ae8ff0 at consistency null
2022-03-22 05:18:26.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1159171673-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:26.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257410809-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c96d553 at consistency null
2022-03-22 05:18:26.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257410809-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:26.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [55992380-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f96ce1e at consistency null
2022-03-22 05:18:26.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [55992380-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:26.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1430682497-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67b38197 at consistency null
2022-03-22 05:18:26.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1430682497-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:27.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752501302-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64e1539e at consistency null
2022-03-22 05:18:27.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752501302-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:27.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487270746-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e273e1c at consistency null
2022-03-22 05:18:27.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487270746-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:28.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [762113351-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37e8e5f8 at consistency null
2022-03-22 05:18:28.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [762113351-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:30.044 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:18:30.086 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-54, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:18:30.095 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-54, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:18:30.095 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-54, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:18:30.101 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-54, inFlight=0, closed=true] closing connection
2022-03-22 05:18:30.102 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 05:18:30.103 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 128000 ms
2022-03-22 05:18:30.103 DEBUG 10 --- [uster-worker-63] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:18:30.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1259344152-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cac65ab at consistency null
2022-03-22 05:18:30.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1259344152-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:30.423 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:18:35.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [891625918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@223fa203 at consistency null
2022-03-22 05:18:35.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [891625918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:35.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1906347961-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cfaf86 at consistency null
2022-03-22 05:18:35.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1906347961-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:35.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [996954655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75784de2 at consistency null
2022-03-22 05:18:35.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [996954655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:35.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1883180219-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a56ffdb at consistency null
2022-03-22 05:18:35.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1883180219-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:36.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821113299-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a67128 at consistency null
2022-03-22 05:18:36.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821113299-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:36.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421756092-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d6d5d85 at consistency null
2022-03-22 05:18:36.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421756092-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:36.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074149677-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4acd1897 at consistency null
2022-03-22 05:18:36.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074149677-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:37.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1221902146-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27088e32 at consistency null
2022-03-22 05:18:37.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1221902146-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:37.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066911464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@111184ca at consistency null
2022-03-22 05:18:37.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066911464-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:38.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037826990-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6876218 at consistency null
2022-03-22 05:18:38.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037826990-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:39.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147342588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@242cdbff at consistency null
2022-03-22 05:18:39.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147342588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:39.044 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:18:42.931 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:18:42.938 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:18:44.055 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703662600-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47b6b856 at consistency null
2022-03-22 05:18:44.056 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703662600-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.094 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [181513601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a9c43be at consistency null
2022-03-22 05:18:44.094 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [181513601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.166 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1170555584-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67b66d57 at consistency null
2022-03-22 05:18:44.166 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1170555584-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.233 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266338592-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2507b17d at consistency null
2022-03-22 05:18:44.233 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266338592-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.293 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879800128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5701c983 at consistency null
2022-03-22 05:18:44.293 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879800128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [860533226-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fb8f16e at consistency null
2022-03-22 05:18:44.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [860533226-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.486 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1559639553-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bb0f51b at consistency null
2022-03-22 05:18:44.487 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1559639553-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401996734-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23243749 at consistency null
2022-03-22 05:18:44.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401996734-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.675 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1647012387-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a30bf7f at consistency null
2022-03-22 05:18:44.675 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1647012387-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.795 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [67654253-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52ab12d6 at consistency null
2022-03-22 05:18:44.795 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [67654253-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:44.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1763912403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e8f2db5 at consistency null
2022-03-22 05:18:44.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1763912403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:45.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1011593084-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c93c152 at consistency null
2022-03-22 05:18:45.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1011593084-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:45.253 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [92844196-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@132750f8 at consistency null
2022-03-22 05:18:45.253 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [92844196-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:45.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129648450-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1910dde8 at consistency null
2022-03-22 05:18:45.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129648450-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:45.676 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [860937801-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ee5a9fa at consistency null
2022-03-22 05:18:45.676 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [860937801-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:45.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1384228624-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c0e7025 at consistency null
2022-03-22 05:18:45.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1384228624-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:46.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315367540-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ae73719 at consistency null
2022-03-22 05:18:46.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315367540-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:46.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046356590-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69eeedc6 at consistency null
2022-03-22 05:18:46.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046356590-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:46.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1290786702-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66ee2786 at consistency null
2022-03-22 05:18:46.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1290786702-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:46.769 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [327766052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@978eda8 at consistency null
2022-03-22 05:18:46.769 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [327766052-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:47.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1622482337-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a544c61 at consistency null
2022-03-22 05:18:47.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1622482337-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:47.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1586497093-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@644a1e30 at consistency null
2022-03-22 05:18:47.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1586497093-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:47.431 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:18:52.539 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1819408830-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@738fc728 at consistency null
2022-03-22 05:18:52.540 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1819408830-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:52.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [891329636-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d54615c at consistency null
2022-03-22 05:18:52.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [891329636-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:52.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100348788-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65bb15d3 at consistency null
2022-03-22 05:18:52.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100348788-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:52.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [592204985-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51b42cca at consistency null
2022-03-22 05:18:52.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [592204985-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:52.871 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083434343-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c8f1753 at consistency null
2022-03-22 05:18:52.871 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083434343-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:53.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1910608557-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@424fd31f at consistency null
2022-03-22 05:18:53.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1910608557-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:53.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [177332354-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f674308 at consistency null
2022-03-22 05:18:53.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [177332354-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:53.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1262060789-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f45e252 at consistency null
2022-03-22 05:18:53.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1262060789-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:53.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [341768232-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bd44966 at consistency null
2022-03-22 05:18:53.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [341768232-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:54.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1775727860-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@173fb028 at consistency null
2022-03-22 05:18:54.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1775727860-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:55.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1816595644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5205cec4 at consistency null
2022-03-22 05:18:55.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1816595644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:18:55.016 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:19:00.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801331693-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72f2e1a4 at consistency null
2022-03-22 05:19:00.047 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801331693-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:00.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136452374-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7be419e5 at consistency null
2022-03-22 05:19:00.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136452374-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:00.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1509997618-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11b05841 at consistency null
2022-03-22 05:19:00.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1509997618-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:00.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [251033890-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6dbfd75d at consistency null
2022-03-22 05:19:00.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [251033890-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:00.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [346907226-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@825f48f at consistency null
2022-03-22 05:19:00.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [346907226-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:00.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [207565680-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5355ff5f at consistency null
2022-03-22 05:19:00.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [207565680-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:01.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407315941-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4034c724 at consistency null
2022-03-22 05:19:01.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407315941-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:02.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860412370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5322e540 at consistency null
2022-03-22 05:19:02.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860412370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:03.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [295941094-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7404f227 at consistency null
2022-03-22 05:19:03.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [295941094-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:03.729 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:19:08.744 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1678951953-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8d754ff at consistency null
2022-03-22 05:19:08.744 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1678951953-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:08.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [944804785-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ebcbf26 at consistency null
2022-03-22 05:19:08.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [944804785-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:08.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1264470618-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a705f54 at consistency null
2022-03-22 05:19:08.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1264470618-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:08.988 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1198455892-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79b615f7 at consistency null
2022-03-22 05:19:08.988 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1198455892-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:09.067 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [901683430-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b74d692 at consistency null
2022-03-22 05:19:09.067 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [901683430-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:09.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156195067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40ba1555 at consistency null
2022-03-22 05:19:09.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156195067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:09.255 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [821747474-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@681878e9 at consistency null
2022-03-22 05:19:09.255 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [821747474-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:09.374 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [4589829-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2148d1b0 at consistency null
2022-03-22 05:19:09.374 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [4589829-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:09.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098831939-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c65434d at consistency null
2022-03-22 05:19:09.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098831939-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:10.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839243799-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c65e729 at consistency null
2022-03-22 05:19:10.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839243799-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:10.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1424065290-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7791037c at consistency null
2022-03-22 05:19:10.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1424065290-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:12.481 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692378802-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10b3996f at consistency null
2022-03-22 05:19:12.481 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692378802-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:12.484 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:19:12.962 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:19:12.978 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:19:17.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406457514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76aadf1f at consistency null
2022-03-22 05:19:17.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406457514-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:17.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1334667687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@585324c at consistency null
2022-03-22 05:19:17.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1334667687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:17.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388778639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7af04b63 at consistency null
2022-03-22 05:19:17.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388778639-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:17.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [241145070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7684ddb3 at consistency null
2022-03-22 05:19:17.949 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [241145070-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:18.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2068537865-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5676a65b at consistency null
2022-03-22 05:19:18.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2068537865-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:18.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1605364886-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7acdf98b at consistency null
2022-03-22 05:19:18.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1605364886-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:19.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [715308510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29e0ee2 at consistency null
2022-03-22 05:19:19.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [715308510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:20.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1479062656-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52a73247 at consistency null
2022-03-22 05:19:20.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1479062656-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:20.407 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:19:25.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027151014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31801236 at consistency null
2022-03-22 05:19:25.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027151014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:25.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1488471670-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2494a42c at consistency null
2022-03-22 05:19:25.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1488471670-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:25.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1732992242-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@560f4742 at consistency null
2022-03-22 05:19:25.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1732992242-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:25.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052745263-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dd162ce at consistency null
2022-03-22 05:19:25.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052745263-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:25.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479903532-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c71d6e0 at consistency null
2022-03-22 05:19:25.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479903532-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:26.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [101433663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@dc5b686 at consistency null
2022-03-22 05:19:26.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [101433663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:26.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [999989696-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@215cc52b at consistency null
2022-03-22 05:19:26.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [999989696-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:26.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [313044364-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@172d14dc at consistency null
2022-03-22 05:19:26.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [313044364-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:26.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1526862634-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15e9160f at consistency null
2022-03-22 05:19:26.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1526862634-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:27.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [56111948-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3538f3b2 at consistency null
2022-03-22 05:19:27.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [56111948-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:27.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1830950470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@728a1da3 at consistency null
2022-03-22 05:19:27.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1830950470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:27.934 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:19:33.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [984683171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@dfe7cda at consistency null
2022-03-22 05:19:33.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [984683171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:33.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1910623644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@507a4a04 at consistency null
2022-03-22 05:19:33.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1910623644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:33.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862012095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@222d96da at consistency null
2022-03-22 05:19:33.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862012095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:33.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1317855923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e8f039 at consistency null
2022-03-22 05:19:33.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1317855923-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:33.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2082133451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e0e5452 at consistency null
2022-03-22 05:19:33.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2082133451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:33.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [57178873-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c261bfd at consistency null
2022-03-22 05:19:33.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [57178873-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:33.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [983934097-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32f3c9e5 at consistency null
2022-03-22 05:19:33.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [983934097-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:33.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [328548364-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@755d9eba at consistency null
2022-03-22 05:19:33.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [328548364-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:33.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [965438264-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79fde887 at consistency null
2022-03-22 05:19:33.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [965438264-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:33.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046058818-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3548fdb6 at consistency null
2022-03-22 05:19:33.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046058818-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:34.245 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851970497-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1958b61b at consistency null
2022-03-22 05:19:34.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851970497-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:34.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1473149729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e953c4 at consistency null
2022-03-22 05:19:34.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1473149729-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:35.160 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897456203-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16463a8e at consistency null
2022-03-22 05:19:35.160 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897456203-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:36.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1876591063-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6411d0bd at consistency null
2022-03-22 05:19:36.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1876591063-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:36.846 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [400472886-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38254a4a at consistency null
2022-03-22 05:19:36.846 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [400472886-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:36.849 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:19:41.862 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862838175-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@721849f9 at consistency null
2022-03-22 05:19:41.863 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862838175-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:41.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989425145-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55ad9f7e at consistency null
2022-03-22 05:19:41.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989425145-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:41.989 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1299956933-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f53394e at consistency null
2022-03-22 05:19:41.990 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1299956933-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:42.047 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [798937431-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d3653f4 at consistency null
2022-03-22 05:19:42.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [798937431-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:42.158 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1925345310-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dd1802a at consistency null
2022-03-22 05:19:42.158 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1925345310-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:42.272 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633256512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24beaa81 at consistency null
2022-03-22 05:19:42.273 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633256512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:42.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921706462-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2816e11 at consistency null
2022-03-22 05:19:42.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921706462-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:42.500 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1866298430-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1db73446 at consistency null
2022-03-22 05:19:42.500 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1866298430-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:42.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245283655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c76270a at consistency null
2022-03-22 05:19:42.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245283655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:42.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587706990-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dcd7ea7 at consistency null
2022-03-22 05:19:42.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587706990-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:42.979 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:19:42.994 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:19:43.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754957451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@543572e9 at consistency null
2022-03-22 05:19:43.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754957451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:43.438 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820492578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7004792d at consistency null
2022-03-22 05:19:43.438 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820492578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:43.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [398281078-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d775e50 at consistency null
2022-03-22 05:19:43.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [398281078-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:44.260 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1513162467-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bf9391f at consistency null
2022-03-22 05:19:44.267 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1513162467-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:44.689 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721986544-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@424aeb3 at consistency null
2022-03-22 05:19:44.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721986544-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:45.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [507541909-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@573a720b at consistency null
2022-03-22 05:19:45.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [507541909-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:45.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1665885650-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20c83362 at consistency null
2022-03-22 05:19:45.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1665885650-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:45.510 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:19:50.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [824144198-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@333bb269 at consistency null
2022-03-22 05:19:50.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [824144198-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:50.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1278105208-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3795342f at consistency null
2022-03-22 05:19:50.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1278105208-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:50.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1363088057-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5627429c at consistency null
2022-03-22 05:19:50.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1363088057-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:51.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1768065209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a9d5102 at consistency null
2022-03-22 05:19:51.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1768065209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:51.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [591221138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74db10c2 at consistency null
2022-03-22 05:19:51.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [591221138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:52.444 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [488525339-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a71663a at consistency null
2022-03-22 05:19:52.444 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [488525339-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:54.054 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [874331611-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6aeea4a at consistency null
2022-03-22 05:19:54.054 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [874331611-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:54.056 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:19:59.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1229409441-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76cbbedc at consistency null
2022-03-22 05:19:59.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1229409441-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:59.166 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12527066-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a3aedc7 at consistency null
2022-03-22 05:19:59.166 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12527066-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:59.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1866386752-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b1b000c at consistency null
2022-03-22 05:19:59.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1866386752-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:59.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [848352921-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a96519f at consistency null
2022-03-22 05:19:59.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [848352921-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:59.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1094825518-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61802b4f at consistency null
2022-03-22 05:19:59.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1094825518-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:19:59.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [950651433-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cd9b66d at consistency null
2022-03-22 05:19:59.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [950651433-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:00.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [560219313-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fa1d09a at consistency null
2022-03-22 05:20:00.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [560219313-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:00.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [368343578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77c37258 at consistency null
2022-03-22 05:20:00.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [368343578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:01.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506303564-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17f9e5bc at consistency null
2022-03-22 05:20:01.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506303564-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:02.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [801517534-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@371670bc at consistency null
2022-03-22 05:20:02.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [801517534-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:02.529 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:20:07.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [746895195-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bab85b4 at consistency null
2022-03-22 05:20:07.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [746895195-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:07.597 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [446049403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@272eb12f at consistency null
2022-03-22 05:20:07.597 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [446049403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:07.627 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [28785499-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23b9b6e6 at consistency null
2022-03-22 05:20:07.627 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [28785499-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:07.664 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [529615980-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24381100 at consistency null
2022-03-22 05:20:07.664 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [529615980-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:07.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371596970-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dddd660 at consistency null
2022-03-22 05:20:07.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371596970-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:07.753 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1039666657-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f28ab3d at consistency null
2022-03-22 05:20:07.753 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1039666657-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:07.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [95261926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21c83ab8 at consistency null
2022-03-22 05:20:07.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [95261926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:07.856 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [281696787-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@459c9f90 at consistency null
2022-03-22 05:20:07.856 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [281696787-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:07.961 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [918537710-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36813f95 at consistency null
2022-03-22 05:20:07.962 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [918537710-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:08.196 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1263878621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c7a1b3b at consistency null
2022-03-22 05:20:08.196 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1263878621-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:08.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [582416117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c07fbf6 at consistency null
2022-03-22 05:20:08.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [582416117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:08.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626221170-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d468ff9 at consistency null
2022-03-22 05:20:08.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626221170-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:09.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674628404-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42846a53 at consistency null
2022-03-22 05:20:09.041 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674628404-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:09.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1225585734-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ecb12fb at consistency null
2022-03-22 05:20:09.450 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1225585734-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:09.873 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553507020-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d70849f at consistency null
2022-03-22 05:20:09.873 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553507020-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:10.289 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1295302295-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@400f38f at consistency null
2022-03-22 05:20:10.290 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1295302295-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:11.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1434659750-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6537d7ae at consistency null
2022-03-22 05:20:11.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1434659750-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:11.104 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:20:13.006 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:20:13.010 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:20:16.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063813524-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@131d5d39 at consistency null
2022-03-22 05:20:16.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063813524-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:16.142 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1653868794-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a8067b9 at consistency null
2022-03-22 05:20:16.142 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1653868794-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:16.200 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171754041-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@591f7bed at consistency null
2022-03-22 05:20:16.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171754041-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:16.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1185810115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ce5dc7 at consistency null
2022-03-22 05:20:16.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1185810115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:16.270 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [663796329-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b433c32 at consistency null
2022-03-22 05:20:16.270 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [663796329-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:16.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1702999481-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22a7214c at consistency null
2022-03-22 05:20:16.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1702999481-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:16.369 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155225658-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c5f6e6 at consistency null
2022-03-22 05:20:16.369 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155225658-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:16.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1072564687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7095a35c at consistency null
2022-03-22 05:20:16.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1072564687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:16.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487571369-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1255b3d0 at consistency null
2022-03-22 05:20:16.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487571369-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:17.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1976303052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9e34d97 at consistency null
2022-03-22 05:20:17.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1976303052-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:17.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029493265-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@263ee88b at consistency null
2022-03-22 05:20:17.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029493265-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:18.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [924336817-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bec2053 at consistency null
2022-03-22 05:20:18.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [924336817-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:18.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1460289804-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4418d7cd at consistency null
2022-03-22 05:20:18.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1460289804-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:18.916 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:20:23.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2097537980-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5704a52c at consistency null
2022-03-22 05:20:23.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2097537980-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:24.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [970391321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6af75f7 at consistency null
2022-03-22 05:20:24.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [970391321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:24.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1278099288-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c5b23c9 at consistency null
2022-03-22 05:20:24.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1278099288-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:24.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235813872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e7f3930 at consistency null
2022-03-22 05:20:24.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235813872-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:24.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [459478906-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ca0c216 at consistency null
2022-03-22 05:20:24.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [459478906-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:24.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365108536-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11dd02b8 at consistency null
2022-03-22 05:20:24.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365108536-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:24.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [682243111-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@545d2e21 at consistency null
2022-03-22 05:20:24.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [682243111-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:24.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [690823751-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@338c3b23 at consistency null
2022-03-22 05:20:24.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [690823751-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:25.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [712386783-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5956ef36 at consistency null
2022-03-22 05:20:25.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [712386783-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:25.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [659675764-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2067fe66 at consistency null
2022-03-22 05:20:25.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [659675764-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:26.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587796405-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d76f06d at consistency null
2022-03-22 05:20:26.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587796405-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:27.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [509404988-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ff185c3 at consistency null
2022-03-22 05:20:27.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [509404988-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:27.110 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:20:32.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [100074744-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21b0ec54 at consistency null
2022-03-22 05:20:32.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [100074744-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:32.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752675912-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e3ba2ca at consistency null
2022-03-22 05:20:32.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752675912-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:32.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1819473431-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1464d8f8 at consistency null
2022-03-22 05:20:32.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1819473431-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:32.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [367954338-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57f8982a at consistency null
2022-03-22 05:20:32.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [367954338-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:32.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671325987-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@710c0ccd at consistency null
2022-03-22 05:20:32.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671325987-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:33.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1668712647-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1242ddd4 at consistency null
2022-03-22 05:20:33.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1668712647-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:33.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [416225520-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20d3647c at consistency null
2022-03-22 05:20:33.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [416225520-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:33.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2030907858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33799184 at consistency null
2022-03-22 05:20:33.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2030907858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:34.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1304646487-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22028942 at consistency null
2022-03-22 05:20:34.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1304646487-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:34.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [447007384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26aa4cab at consistency null
2022-03-22 05:20:34.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [447007384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:35.650 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2051819940-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1de12c86 at consistency null
2022-03-22 05:20:35.651 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2051819940-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:35.666 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:20:38.103 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:20:38.122 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-55, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:20:38.122 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-55, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:20:38.122 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-55, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:20:38.128 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-55, inFlight=0, closed=true] closing connection
2022-03-22 05:20:38.129 DEBUG 10 --- [uster-worker-64] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:20:38.130 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 256000 milliseconds
2022-03-22 05:20:38.130 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 256000 ms
2022-03-22 05:20:40.678 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1784636574-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@626b6b0f at consistency null
2022-03-22 05:20:40.678 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1784636574-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:40.745 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1037159273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5495b699 at consistency null
2022-03-22 05:20:40.745 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1037159273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:40.868 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [27080257-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20ae77b4 at consistency null
2022-03-22 05:20:40.868 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [27080257-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:41.086 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [428101311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d35cca0 at consistency null
2022-03-22 05:20:41.087 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [428101311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:41.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [149860997-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@746d92e1 at consistency null
2022-03-22 05:20:41.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [149860997-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:41.940 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1003989618-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c8800e3 at consistency null
2022-03-22 05:20:41.940 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1003989618-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:42.373 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401994359-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f7b811e at consistency null
2022-03-22 05:20:42.374 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401994359-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:42.797 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1759179020-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5799e88a at consistency null
2022-03-22 05:20:42.797 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1759179020-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:43.012 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:20:43.018 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:20:43.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [358419464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a392f56 at consistency null
2022-03-22 05:20:43.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [358419464-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:43.644 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [325038315-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@361abbea at consistency null
2022-03-22 05:20:43.644 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [325038315-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:44.457 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1174066675-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25faaf46 at consistency null
2022-03-22 05:20:44.457 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1174066675-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:44.459 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:20:49.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2056724177-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dbfcecb at consistency null
2022-03-22 05:20:49.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2056724177-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:49.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123166919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e2f05b5 at consistency null
2022-03-22 05:20:49.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123166919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:49.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776369710-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d892671 at consistency null
2022-03-22 05:20:49.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776369710-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:50.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049394092-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7df93819 at consistency null
2022-03-22 05:20:50.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049394092-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:50.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [948223407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e7378fb at consistency null
2022-03-22 05:20:50.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [948223407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:50.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82420512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2867c02d at consistency null
2022-03-22 05:20:50.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82420512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:51.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1743414179-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c99c1fe at consistency null
2022-03-22 05:20:51.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1743414179-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:53.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [664813783-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58941584 at consistency null
2022-03-22 05:20:53.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [664813783-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:53.431 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:20:58.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [978643017-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d051acd at consistency null
2022-03-22 05:20:58.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [978643017-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:58.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936567424-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bf5cbe8 at consistency null
2022-03-22 05:20:58.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936567424-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:58.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202597429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49416159 at consistency null
2022-03-22 05:20:58.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202597429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:58.740 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1647147247-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@310165c0 at consistency null
2022-03-22 05:20:58.740 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1647147247-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:59.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2133246531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6407c25f at consistency null
2022-03-22 05:20:59.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2133246531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:59.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630627979-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2168ffbc at consistency null
2022-03-22 05:20:59.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630627979-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:20:59.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1473789214-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63611961 at consistency null
2022-03-22 05:20:59.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1473789214-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:00.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1580981514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59745fa8 at consistency null
2022-03-22 05:21:00.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1580981514-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:02.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390477908-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bb7be3 at consistency null
2022-03-22 05:21:02.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390477908-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:02.207 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:21:07.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2111405831-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6548e137 at consistency null
2022-03-22 05:21:07.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2111405831-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:07.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1803525487-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e8946ee at consistency null
2022-03-22 05:21:07.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1803525487-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:07.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1798098567-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75fbc188 at consistency null
2022-03-22 05:21:07.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1798098567-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:07.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1301204782-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2dc7c8b7 at consistency null
2022-03-22 05:21:07.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1301204782-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:08.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2073669855-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f24d561 at consistency null
2022-03-22 05:21:08.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2073669855-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:08.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1822088742-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b5e4c5d at consistency null
2022-03-22 05:21:08.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1822088742-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:09.243 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1632599492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6002333f at consistency null
2022-03-22 05:21:09.244 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1632599492-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:10.855 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1510156668-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7393ddf8 at consistency null
2022-03-22 05:21:10.855 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1510156668-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:10.861 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:21:13.041 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:21:13.045 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:21:15.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1814721963-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63300299 at consistency null
2022-03-22 05:21:15.875 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1814721963-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:15.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1459355110-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@125a4e0b at consistency null
2022-03-22 05:21:15.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1459355110-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:15.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1287658400-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f4b2da5 at consistency null
2022-03-22 05:21:15.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1287658400-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:16.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264999821-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9813f5b at consistency null
2022-03-22 05:21:16.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264999821-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:16.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [812857586-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ac6c038 at consistency null
2022-03-22 05:21:16.079 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [812857586-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:16.193 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [963569623-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a2ce17d at consistency null
2022-03-22 05:21:16.193 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [963569623-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:16.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1967121985-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cd75aa1 at consistency null
2022-03-22 05:21:16.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1967121985-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:16.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [641418384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50cc8c03 at consistency null
2022-03-22 05:21:16.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [641418384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:17.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254172173-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1da6a86e at consistency null
2022-03-22 05:21:17.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254172173-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:18.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1094206049-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77dcbc45 at consistency null
2022-03-22 05:21:18.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1094206049-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:18.524 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:21:23.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1270856169-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41e8e24d at consistency null
2022-03-22 05:21:23.547 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1270856169-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:23.626 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [928803786-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7570d2e7 at consistency null
2022-03-22 05:21:23.626 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [928803786-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:23.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [170364696-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dbde1ff at consistency null
2022-03-22 05:21:23.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [170364696-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:23.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [152039287-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f114099 at consistency null
2022-03-22 05:21:23.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [152039287-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:23.892 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:21:23.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-32, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:21:23.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-32, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:21:23.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-32, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:21:23.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-32, inFlight=0, closed=true] closing connection
2022-03-22 05:21:23.916 DEBUG 10 --- [uster-worker-65] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:21:23.917 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 512000 milliseconds
2022-03-22 05:21:23.917 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 512000 ms
2022-03-22 05:21:24.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070165838-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4da91d3b at consistency null
2022-03-22 05:21:24.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070165838-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:24.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1805599046-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6da96dfb at consistency null
2022-03-22 05:21:24.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1805599046-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:24.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576422498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a7b725a at consistency null
2022-03-22 05:21:24.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576422498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:25.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [737441020-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fbeeceb at consistency null
2022-03-22 05:21:25.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [737441020-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:25.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1605619155-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@795e9452 at consistency null
2022-03-22 05:21:25.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1605619155-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:26.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1312236050-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1eb24036 at consistency null
2022-03-22 05:21:26.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1312236050-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:27.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [567141098-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f5a1818 at consistency null
2022-03-22 05:21:27.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [567141098-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:27.304 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:21:32.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [697704891-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@372b66ac at consistency null
2022-03-22 05:21:32.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [697704891-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:32.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1753918479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a5a981f at consistency null
2022-03-22 05:21:32.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1753918479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:32.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [526001342-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36e5cc92 at consistency null
2022-03-22 05:21:32.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [526001342-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:32.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1372475149-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7930988e at consistency null
2022-03-22 05:21:32.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1372475149-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:32.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1238181750-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15429f02 at consistency null
2022-03-22 05:21:32.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1238181750-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:32.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1956249748-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bb677d5 at consistency null
2022-03-22 05:21:32.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1956249748-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:32.948 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [975938925-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63afb56d at consistency null
2022-03-22 05:21:32.948 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [975938925-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:33.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431891202-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e26a57e at consistency null
2022-03-22 05:21:33.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431891202-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:33.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [971526612-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f1d9337 at consistency null
2022-03-22 05:21:33.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [971526612-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:33.282 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [500328669-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@575d57de at consistency null
2022-03-22 05:21:33.283 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [500328669-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:33.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2050340551-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49479485 at consistency null
2022-03-22 05:21:33.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2050340551-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:33.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [318450864-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6261d586 at consistency null
2022-03-22 05:21:33.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [318450864-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:33.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776728522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@119666d6 at consistency null
2022-03-22 05:21:33.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776728522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:33.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736220843-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c48309d at consistency null
2022-03-22 05:21:33.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736220843-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:33.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [807969037-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@595e7a2d at consistency null
2022-03-22 05:21:33.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [807969037-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:34.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2071823636-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@357f5965 at consistency null
2022-03-22 05:21:34.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2071823636-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:34.445 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [654920536-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17fd39ca at consistency null
2022-03-22 05:21:34.445 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [654920536-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:34.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636669674-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7adc3f70 at consistency null
2022-03-22 05:21:34.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636669674-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:35.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344746384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50f9ad14 at consistency null
2022-03-22 05:21:35.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344746384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:35.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1206208933-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f02721f at consistency null
2022-03-22 05:21:35.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1206208933-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:36.141 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [719030666-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@614f3b5c at consistency null
2022-03-22 05:21:36.141 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [719030666-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:36.143 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:21:41.154 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [426608549-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7af3bfba at consistency null
2022-03-22 05:21:41.154 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [426608549-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.192 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1891605675-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@677f2cfd at consistency null
2022-03-22 05:21:41.193 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1891605675-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [502942584-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b613887 at consistency null
2022-03-22 05:21:41.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [502942584-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.263 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1796933268-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@736c6b6c at consistency null
2022-03-22 05:21:41.264 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1796933268-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [515831037-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@148504d4 at consistency null
2022-03-22 05:21:41.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [515831037-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.391 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [317225285-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a03ca7d at consistency null
2022-03-22 05:21:41.391 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [317225285-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.495 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1569079787-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68446eba at consistency null
2022-03-22 05:21:41.495 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1569079787-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1847247721-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78634fd5 at consistency null
2022-03-22 05:21:41.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1847247721-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [124443467-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@447c57a9 at consistency null
2022-03-22 05:21:41.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [124443467-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1452914866-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50382d79 at consistency null
2022-03-22 05:21:41.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1452914866-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:41.952 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [964633235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bc2a87a at consistency null
2022-03-22 05:21:41.952 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [964633235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:42.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [21409630-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@793d4130 at consistency null
2022-03-22 05:21:42.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [21409630-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:42.167 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344773310-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f5a90a1 at consistency null
2022-03-22 05:21:42.167 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344773310-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:42.374 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273033248-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e983dd6 at consistency null
2022-03-22 05:21:42.374 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273033248-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:42.788 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [189475445-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4097cd07 at consistency null
2022-03-22 05:21:42.789 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [189475445-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:43.069 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:21:43.086 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:21:43.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1061378077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@533020e4 at consistency null
2022-03-22 05:21:43.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1061378077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:44.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399709991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58e46dbb at consistency null
2022-03-22 05:21:44.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399709991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:44.034 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:21:49.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [525363352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@571d9dfc at consistency null
2022-03-22 05:21:49.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [525363352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:49.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1302000005-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@299c8ded at consistency null
2022-03-22 05:21:49.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1302000005-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:49.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757207451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b79af3a at consistency null
2022-03-22 05:21:49.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757207451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:49.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1424255858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8b9c5b at consistency null
2022-03-22 05:21:49.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1424255858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:49.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215094238-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b4213cb at consistency null
2022-03-22 05:21:49.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215094238-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:50.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1382026127-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76fc02ab at consistency null
2022-03-22 05:21:50.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1382026127-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:50.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552839259-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@404a7331 at consistency null
2022-03-22 05:21:50.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552839259-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:50.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [726023491-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c7f4627 at consistency null
2022-03-22 05:21:50.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [726023491-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:51.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [90871768-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3788c228 at consistency null
2022-03-22 05:21:51.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [90871768-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:51.523 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:21:56.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1468319881-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21f76728 at consistency null
2022-03-22 05:21:56.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1468319881-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:56.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025288840-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32b74bdc at consistency null
2022-03-22 05:21:56.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025288840-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:56.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644294904-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@549f6dd3 at consistency null
2022-03-22 05:21:56.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644294904-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:57.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855932215-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25f76cea at consistency null
2022-03-22 05:21:57.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855932215-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:57.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1119169374-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fa3815b at consistency null
2022-03-22 05:21:57.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1119169374-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:58.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23401652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3742d4d8 at consistency null
2022-03-22 05:21:58.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23401652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:59.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1822339154-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@189cc616 at consistency null
2022-03-22 05:21:59.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1822339154-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:59.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095289651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69279d3 at consistency null
2022-03-22 05:21:59.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095289651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:21:59.930 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:22:04.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [392674858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25988903 at consistency null
2022-03-22 05:22:04.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [392674858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1049548998-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dd6d7c at consistency null
2022-03-22 05:22:05.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1049548998-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [655094629-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a0fb787 at consistency null
2022-03-22 05:22:05.092 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [655094629-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.181 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1201778874-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fc605f4 at consistency null
2022-03-22 05:22:05.182 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1201778874-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.243 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1839059082-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e77599e at consistency null
2022-03-22 05:22:05.243 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1839059082-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622689559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@612e73a1 at consistency null
2022-03-22 05:22:05.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622689559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.366 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576346122-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@669a438c at consistency null
2022-03-22 05:22:05.366 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576346122-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [387677550-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39ac030e at consistency null
2022-03-22 05:22:05.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [387677550-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.491 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344057479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d23060c at consistency null
2022-03-22 05:22:05.491 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344057479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.549 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1513800901-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f82df72 at consistency null
2022-03-22 05:22:05.549 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1513800901-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [782977274-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bb0692d at consistency null
2022-03-22 05:22:05.657 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [782977274-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:05.867 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1654104140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72ddab26 at consistency null
2022-03-22 05:22:05.868 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1654104140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:06.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2042218311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@675bc017 at consistency null
2022-03-22 05:22:06.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2042218311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:07.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1974020185-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6eabfac1 at consistency null
2022-03-22 05:22:07.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1974020185-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:08.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [28394126-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46fa1f38 at consistency null
2022-03-22 05:22:08.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [28394126-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:08.753 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:22:13.098 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:22:13.112 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:22:13.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1593513780-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5961085a at consistency null
2022-03-22 05:22:13.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1593513780-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:13.881 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1766813779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60832019 at consistency null
2022-03-22 05:22:13.881 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1766813779-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:13.988 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1427662531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@514fb33a at consistency null
2022-03-22 05:22:13.988 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1427662531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:14.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1580661724-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47da65f4 at consistency null
2022-03-22 05:22:14.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1580661724-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:14.644 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1483871641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16a8868d at consistency null
2022-03-22 05:22:14.644 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1483871641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:15.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1244474156-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fd2ffa9 at consistency null
2022-03-22 05:22:15.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1244474156-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:16.289 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553298132-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4328470a at consistency null
2022-03-22 05:22:16.290 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553298132-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:16.305 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:22:21.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890542721-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57c97f5e at consistency null
2022-03-22 05:22:21.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890542721-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:21.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2105708391-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58707c4 at consistency null
2022-03-22 05:22:21.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2105708391-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:21.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1591062448-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d508f6a at consistency null
2022-03-22 05:22:21.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1591062448-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:21.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [698443116-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c4187dd at consistency null
2022-03-22 05:22:21.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [698443116-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:21.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [946872907-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@606e3484 at consistency null
2022-03-22 05:22:21.861 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [946872907-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:22.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1365277589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d6aa66a at consistency null
2022-03-22 05:22:22.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1365277589-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:22.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [582743590-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24b84972 at consistency null
2022-03-22 05:22:22.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [582743590-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:22.566 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [9665816-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54b75d57 at consistency null
2022-03-22 05:22:22.566 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [9665816-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:23.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969438523-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@456c8951 at consistency null
2022-03-22 05:22:23.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969438523-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:23.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797655876-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dbd2e60 at consistency null
2022-03-22 05:22:23.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797655876-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:24.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [382852288-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ba81d7c at consistency null
2022-03-22 05:22:24.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [382852288-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:25.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229710015-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@451ba0bb at consistency null
2022-03-22 05:22:25.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229710015-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:25.203 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:22:30.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1999295903-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@641c9f48 at consistency null
2022-03-22 05:22:30.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1999295903-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:30.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [750627347-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40f66385 at consistency null
2022-03-22 05:22:30.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [750627347-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:30.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873612440-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bdec51a at consistency null
2022-03-22 05:22:30.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873612440-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:30.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [837011013-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33d8b6a2 at consistency null
2022-03-22 05:22:30.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [837011013-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:30.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [281308107-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@333e994e at consistency null
2022-03-22 05:22:30.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [281308107-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:30.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1374201586-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40a72ee7 at consistency null
2022-03-22 05:22:30.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1374201586-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:31.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012749523-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d0cd5fc at consistency null
2022-03-22 05:22:31.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012749523-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:31.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1476800391-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b170d59 at consistency null
2022-03-22 05:22:31.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1476800391-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:31.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1798278527-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ca40d9c at consistency null
2022-03-22 05:22:31.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1798278527-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:32.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2145679838-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@558ddef9 at consistency null
2022-03-22 05:22:32.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2145679838-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:33.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1214115447-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ed54281 at consistency null
2022-03-22 05:22:33.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1214115447-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:33.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [558795500-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dfbefc7 at consistency null
2022-03-22 05:22:33.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [558795500-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:33.930 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:22:38.937 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916307919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71bfd509 at consistency null
2022-03-22 05:22:38.937 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916307919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:38.980 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1785399883-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4afeb7d1 at consistency null
2022-03-22 05:22:38.980 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1785399883-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:39.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1306689319-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e3b3545 at consistency null
2022-03-22 05:22:39.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1306689319-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:39.089 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [414094749-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c9c6e92 at consistency null
2022-03-22 05:22:39.089 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [414094749-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:39.195 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1133912704-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cfb8a1b at consistency null
2022-03-22 05:22:39.196 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1133912704-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:39.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1906135531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bfd0ddc at consistency null
2022-03-22 05:22:39.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1906135531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:39.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [794301368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@449a02cf at consistency null
2022-03-22 05:22:39.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [794301368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:39.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136028083-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@160388 at consistency null
2022-03-22 05:22:39.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136028083-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:39.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [826724711-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3deac3 at consistency null
2022-03-22 05:22:39.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [826724711-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:39.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [709876909-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@238d94f5 at consistency null
2022-03-22 05:22:39.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [709876909-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:39.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1164304271-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37956631 at consistency null
2022-03-22 05:22:39.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1164304271-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:40.349 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [904846689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7af3908 at consistency null
2022-03-22 05:22:40.350 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [904846689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:40.754 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [849674066-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@682bb0cc at consistency null
2022-03-22 05:22:40.754 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [849674066-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:41.172 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1725626143-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@274eab00 at consistency null
2022-03-22 05:22:41.173 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1725626143-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:41.579 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220775916-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bf6e536 at consistency null
2022-03-22 05:22:41.579 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220775916-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:41.990 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315837181-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@456ffc39 at consistency null
2022-03-22 05:22:41.990 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315837181-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:42.393 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1537551373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70f79988 at consistency null
2022-03-22 05:22:42.394 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1537551373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:42.398 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:22:43.116 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:22:43.134 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:22:47.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1786141289-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22b33a3d at consistency null
2022-03-22 05:22:47.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1786141289-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:47.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [557512046-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ee0a408 at consistency null
2022-03-22 05:22:47.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [557512046-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:47.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533578824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14d41bd9 at consistency null
2022-03-22 05:22:47.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533578824-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:47.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845118712-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58a3eb12 at consistency null
2022-03-22 05:22:47.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845118712-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:47.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [898100284-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63fa40e9 at consistency null
2022-03-22 05:22:47.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [898100284-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:48.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138818663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4811787a at consistency null
2022-03-22 05:22:48.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138818663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:48.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [847154647-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c2fb872 at consistency null
2022-03-22 05:22:48.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [847154647-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:48.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [227332326-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b13279e at consistency null
2022-03-22 05:22:48.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [227332326-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:49.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123743079-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@adfd51d at consistency null
2022-03-22 05:22:49.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123743079-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:50.544 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136127530-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@494eae94 at consistency null
2022-03-22 05:22:50.544 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136127530-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:51.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [98681790-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@482363e7 at consistency null
2022-03-22 05:22:51.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [98681790-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:51.417 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:22:56.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845569540-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6aae82aa at consistency null
2022-03-22 05:22:56.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845569540-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:56.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757158832-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e2dc5df at consistency null
2022-03-22 05:22:56.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757158832-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:56.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344352819-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@222be280 at consistency null
2022-03-22 05:22:56.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344352819-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:56.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [527545646-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71c49179 at consistency null
2022-03-22 05:22:56.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [527545646-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:56.757 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027417548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f61f639 at consistency null
2022-03-22 05:22:56.757 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027417548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:56.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1879977211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f92c537 at consistency null
2022-03-22 05:22:56.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1879977211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:56.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1466828089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62500ba7 at consistency null
2022-03-22 05:22:56.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1466828089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:57.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1168380479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57faf923 at consistency null
2022-03-22 05:22:57.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1168380479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:57.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46808264-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e483596 at consistency null
2022-03-22 05:22:57.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46808264-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:57.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [52605342-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ca700cb at consistency null
2022-03-22 05:22:57.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [52605342-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:57.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [608320270-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b04aaeb at consistency null
2022-03-22 05:22:57.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [608320270-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:22:58.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091756115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bb76349 at consistency null
2022-03-22 05:22:58.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091756115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:00.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [317569221-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@444c092c at consistency null
2022-03-22 05:23:00.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [317569221-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:00.519 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:23:05.569 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72697159-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4069f836 at consistency null
2022-03-22 05:23:05.570 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72697159-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:05.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [838501512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ec44122 at consistency null
2022-03-22 05:23:05.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [838501512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:05.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1732049564-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e89c098 at consistency null
2022-03-22 05:23:05.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1732049564-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:05.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417877294-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7aa8b70c at consistency null
2022-03-22 05:23:05.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417877294-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:05.937 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [918117385-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@758f4827 at consistency null
2022-03-22 05:23:05.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [918117385-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:06.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1392660626-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@695edae5 at consistency null
2022-03-22 05:23:06.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1392660626-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:06.174 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399502964-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79435a56 at consistency null
2022-03-22 05:23:06.174 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399502964-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:06.399 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [154209802-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46ec82a5 at consistency null
2022-03-22 05:23:06.399 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [154209802-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:06.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1964266033-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@492d342f at consistency null
2022-03-22 05:23:06.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1964266033-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:06.858 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [877857473-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@563b461d at consistency null
2022-03-22 05:23:06.866 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [877857473-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:07.094 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855424295-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35492766 at consistency null
2022-03-22 05:23:07.094 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855424295-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:07.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [868000812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@179272c0 at consistency null
2022-03-22 05:23:07.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [868000812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:07.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1684995504-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@373eb051 at consistency null
2022-03-22 05:23:07.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1684995504-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:08.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1221064446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b391305 at consistency null
2022-03-22 05:23:08.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1221064446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:08.724 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:23:13.145 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:23:13.148 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:23:13.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [520453956-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47377113 at consistency null
2022-03-22 05:23:13.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [520453956-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:13.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1229858124-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43b70f47 at consistency null
2022-03-22 05:23:13.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1229858124-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:13.896 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524548264-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c99358 at consistency null
2022-03-22 05:23:13.896 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524548264-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:13.957 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [760852235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@729b095f at consistency null
2022-03-22 05:23:13.957 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [760852235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:14.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [429001403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ed841f6 at consistency null
2022-03-22 05:23:14.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [429001403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:14.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [558770304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@756eba5d at consistency null
2022-03-22 05:23:14.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [558770304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:14.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229209475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3870b4fb at consistency null
2022-03-22 05:23:14.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229209475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:14.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835578058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@128b935b at consistency null
2022-03-22 05:23:14.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835578058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:14.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680255502-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e90b70d at consistency null
2022-03-22 05:23:14.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680255502-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:15.074 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [888968226-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e2c42d0 at consistency null
2022-03-22 05:23:15.075 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [888968226-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:15.889 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1916795475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6346e282 at consistency null
2022-03-22 05:23:15.890 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1916795475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:16.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1525282876-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d19b4fa at consistency null
2022-03-22 05:23:16.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1525282876-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:17.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [189681589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@766a468b at consistency null
2022-03-22 05:23:17.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [189681589-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:17.521 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:23:22.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1539544954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4aecb054 at consistency null
2022-03-22 05:23:22.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1539544954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:22.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1445135155-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1eea0979 at consistency null
2022-03-22 05:23:22.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1445135155-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:22.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134438867-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7eb4c03a at consistency null
2022-03-22 05:23:22.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134438867-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:22.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [987035122-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ace8f79 at consistency null
2022-03-22 05:23:22.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [987035122-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:22.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2104216981-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@750ffa64 at consistency null
2022-03-22 05:23:22.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2104216981-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:23.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866087824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b60847d at consistency null
2022-03-22 05:23:23.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866087824-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:23.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [33363073-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@630d263b at consistency null
2022-03-22 05:23:23.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [33363073-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:23.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [471538217-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d96c864 at consistency null
2022-03-22 05:23:23.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [471538217-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:23.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1069513562-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47c0d768 at consistency null
2022-03-22 05:23:23.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1069513562-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:23.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2004131022-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78bcf081 at consistency null
2022-03-22 05:23:23.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2004131022-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:24.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [507828157-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@720a3d53 at consistency null
2022-03-22 05:23:24.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [507828157-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:24.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1514818574-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9a41e9e at consistency null
2022-03-22 05:23:24.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1514818574-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:24.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510368182-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@489ee355 at consistency null
2022-03-22 05:23:24.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510368182-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:25.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294660697-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57393a3f at consistency null
2022-03-22 05:23:25.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294660697-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:25.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1978350462-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@747a752e at consistency null
2022-03-22 05:23:25.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1978350462-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:25.830 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:23:30.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2101945329-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d83d5d1 at consistency null
2022-03-22 05:23:30.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2101945329-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:31.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1059959751-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6238117f at consistency null
2022-03-22 05:23:31.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1059959751-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:31.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633274439-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c4e37c5 at consistency null
2022-03-22 05:23:31.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633274439-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:31.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [161742661-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bbafe67 at consistency null
2022-03-22 05:23:31.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [161742661-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:31.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1027670827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18d4959c at consistency null
2022-03-22 05:23:31.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1027670827-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:31.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1872768207-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ac5dc10 at consistency null
2022-03-22 05:23:31.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1872768207-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:31.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851489860-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37b83492 at consistency null
2022-03-22 05:23:31.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851489860-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:31.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613366926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bfdadc4 at consistency null
2022-03-22 05:23:31.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613366926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:31.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [291650415-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b9b752 at consistency null
2022-03-22 05:23:31.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [291650415-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:31.951 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2135604184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1388dee9 at consistency null
2022-03-22 05:23:31.951 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2135604184-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:32.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1210008289-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65e274ae at consistency null
2022-03-22 05:23:32.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1210008289-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:32.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [355036192-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3327765b at consistency null
2022-03-22 05:23:32.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [355036192-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:33.640 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [353408169-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@796e444a at consistency null
2022-03-22 05:23:33.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [353408169-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:34.450 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1068394920-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cc4b218 at consistency null
2022-03-22 05:23:34.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1068394920-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:34.452 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:23:39.468 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [219430134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a75f42a at consistency null
2022-03-22 05:23:39.469 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [219430134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:39.499 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1392646424-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5268d491 at consistency null
2022-03-22 05:23:39.499 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1392646424-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:39.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1118206477-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59af8682 at consistency null
2022-03-22 05:23:39.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1118206477-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:39.565 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [243906776-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50f2bbdd at consistency null
2022-03-22 05:23:39.566 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [243906776-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:39.599 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [584504947-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2553d14d at consistency null
2022-03-22 05:23:39.599 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [584504947-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:39.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1633665512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@318b552a at consistency null
2022-03-22 05:23:39.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1633665512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:39.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [274246108-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14b54e0f at consistency null
2022-03-22 05:23:39.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [274246108-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:39.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406442615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51dc3317 at consistency null
2022-03-22 05:23:39.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406442615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:39.940 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1193651504-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a3e59d8 at consistency null
2022-03-22 05:23:39.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1193651504-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:40.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1491760317-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f773b0 at consistency null
2022-03-22 05:23:40.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1491760317-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:40.553 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220803125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f8c90b7 at consistency null
2022-03-22 05:23:40.553 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220803125-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:40.963 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2053710531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f7858da at consistency null
2022-03-22 05:23:40.963 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2053710531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:41.370 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [273632403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17102074 at consistency null
2022-03-22 05:23:41.370 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [273632403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:42.176 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [462134715-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d40bbd5 at consistency null
2022-03-22 05:23:42.176 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [462134715-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:42.178 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:23:43.174 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:23:43.187 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:23:47.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1600634471-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76129fe2 at consistency null
2022-03-22 05:23:47.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1600634471-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:47.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1814814834-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3be1a772 at consistency null
2022-03-22 05:23:47.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1814814834-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:47.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [107610800-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d6433fa at consistency null
2022-03-22 05:23:47.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [107610800-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:47.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1103835523-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d2ab90 at consistency null
2022-03-22 05:23:47.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1103835523-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:48.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [599599963-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67ddd947 at consistency null
2022-03-22 05:23:48.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [599599963-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:48.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270952238-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@513138f9 at consistency null
2022-03-22 05:23:48.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270952238-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:50.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1183815704-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d462ac6 at consistency null
2022-03-22 05:23:50.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1183815704-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:23:50.603 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:15:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:24:09.614 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:16:40Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:24:13.198 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:24:13.202 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:24:43.205 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:24:43.218 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:24:54.133 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:24:54.137 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-56, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:24:54.137 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-56, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:24:54.137 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-56, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:24:54.138 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-56, inFlight=0, closed=true] closing connection
2022-03-22 05:24:54.140 DEBUG 10 --- [uster-worker-66] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:24:54.140 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 512000 milliseconds
2022-03-22 05:24:54.140 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 512000 ms
2022-03-22 05:25:12.456 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:24:59.716Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:25:13.236 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:25:13.286 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:25:43.287 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:25:43.319 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:25:55.517 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:25:44.712Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:26:13.320 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:26:13.323 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:26:43.044 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:26:30.825Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:26:43.364 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:26:43.421 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:26:58.826 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:26:40Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:27:13.422 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:27:13.432 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:27:24.806 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UP /10.233.71.64:9042, scheduling delivery
2022-03-22 05:27:24.806 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] received event UP
2022-03-22 05:27:25.811 DEBUG 10 --- [uster-worker-67] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.233.71.64:9042
2022-03-22 05:27:25.822 DEBUG 10 --- [uster-worker-67] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host UP
2022-03-22 05:27:25.822 DEBUG 10 --- [uster-worker-67] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2022-03-22 05:27:25.822 DEBUG 10 --- [uster-worker-67] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.233.71.64:9042
2022-03-22 05:27:25.822 DEBUG 10 --- [uster-worker-67] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:27:25.830 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-33, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:27:37.890 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-33, inFlight=0, closed=false] because: [/10.233.71.64:9042] Operation timed out
2022-03-22 05:27:37.891 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-33, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:27:37.892 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-33, inFlight=0, closed=true] closing connection
2022-03-22 05:27:37.892 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-33, inFlight=0, closed=true]: there are still pending requests
2022-03-22 05:27:37.894 DEBUG 10 --- [uster-worker-67] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:27:37.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-34, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:27:49.987 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-34, inFlight=0, closed=false] because: [/10.233.71.64:9042] Operation timed out
2022-03-22 05:27:49.988 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-34, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:27:49.988 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-34, inFlight=0, closed=true] closing connection
2022-03-22 05:27:49.989 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-34, inFlight=0, closed=true]: there are still pending requests
2022-03-22 05:27:49.990 WARN 10 --- [ter-timeouter-0] c.d.driver.core.HostConnectionPool : Error creating connection to /10.233.71.64:9042
com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.233.71.64:9042] Operation timed out
at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1496)
at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1575)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 05:27:49.991 DEBUG 10 --- [ter-timeouter-0] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.233.71.64:9042 (1 connections needed, 0 successfully opened)
2022-03-22 05:27:49.991 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Session : Added connection pool for /10.233.71.64:9042
2022-03-22 05:27:50.007 DEBUG 10 --- [uster-worker-68] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] marking host DOWN
2022-03-22 05:27:50.007 DEBUG 10 --- [uster-worker-68] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 05:27:50.008 DEBUG 10 --- [uster-worker-68] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] starting reconnection attempt
2022-03-22 05:27:50.008 DEBUG 10 --- [uster-worker-67] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:27:51.010 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:27:51.027 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-35, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:27:55.833 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:27:55.920 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:28:04.587 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-35, inFlight=0, closed=false] because: [/10.233.71.64:9042] Operation timed out
2022-03-22 05:28:04.587 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-35, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:28:04.588 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-35, inFlight=0, closed=true] closing connection
2022-03-22 05:28:04.588 DEBUG 10 --- [ter-timeouter-0] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.71.64:9042-35, inFlight=0, closed=true]: there are still pending requests
2022-03-22 05:28:04.588 DEBUG 10 --- [uster-worker-68] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:28:04.588 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Operation timed out), scheduling retry in 2000 milliseconds
2022-03-22 05:28:04.588 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 2000 ms
2022-03-22 05:28:06.592 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:28:06.605 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-36, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:28:06.605 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-36, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:28:06.605 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-36, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:28:06.605 DEBUG 10 --- [uster-worker-67] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:28:06.606 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-36, inFlight=0, closed=true] closing connection
2022-03-22 05:28:06.607 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 4000 milliseconds
2022-03-22 05:28:06.608 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 4000 ms
2022-03-22 05:28:07.733 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:28:08.630 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-33, inFlight=0, closed=true] has already terminated
2022-03-22 05:28:10.609 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:28:10.617 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-37, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:28:10.617 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-37, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:28:10.618 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-37, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:28:10.618 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-37, inFlight=0, closed=true] closing connection
2022-03-22 05:28:10.620 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 8000 milliseconds
2022-03-22 05:28:10.620 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 8000 ms
2022-03-22 05:28:10.620 DEBUG 10 --- [uster-worker-68] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:28:17.819 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:28:18.621 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:28:18.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-38, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:28:18.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-38, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:28:18.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-38, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:28:18.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-38, inFlight=0, closed=true] closing connection
2022-03-22 05:28:18.639 DEBUG 10 --- [uster-worker-67] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:28:18.662 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 16000 milliseconds
2022-03-22 05:28:18.662 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 16000 ms
2022-03-22 05:28:21.220 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.233.71.64:9042, scheduling delivery
2022-03-22 05:28:21.221 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] received event DOWN
2022-03-22 05:28:22.222 DEBUG 10 --- [uster-worker-68] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:28:22.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [854052000-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79e320e3 at consistency null
2022-03-22 05:28:22.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [854052000-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:22.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [581246322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e0c8c2c at consistency null
2022-03-22 05:28:22.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [581246322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:23.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1054293044-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51117299 at consistency null
2022-03-22 05:28:23.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1054293044-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:23.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1431912115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@236300f at consistency null
2022-03-22 05:28:23.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1431912115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:23.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1507293812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62b2ec92 at consistency null
2022-03-22 05:28:23.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1507293812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:23.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1402188465-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8b56104 at consistency null
2022-03-22 05:28:23.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1402188465-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:23.631 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-34, inFlight=0, closed=true] has already terminated
2022-03-22 05:28:23.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1411519083-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@526a5f31 at consistency null
2022-03-22 05:28:23.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1411519083-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:24.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1209139962-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@616fab3b at consistency null
2022-03-22 05:28:24.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1209139962-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:24.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720688470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@144c1df8 at consistency null
2022-03-22 05:28:24.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720688470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:25.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1681718403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3752e334 at consistency null
2022-03-22 05:28:25.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1681718403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:25.430 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:28:30.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541657893-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37c1b138 at consistency null
2022-03-22 05:28:30.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541657893-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:30.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2023016452-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b6be288 at consistency null
2022-03-22 05:28:30.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2023016452-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:30.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959197733-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fbc99c0 at consistency null
2022-03-22 05:28:30.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959197733-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:30.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2006830810-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fd79a40 at consistency null
2022-03-22 05:28:30.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2006830810-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:31.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [526770991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4525acf3 at consistency null
2022-03-22 05:28:31.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [526770991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:31.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1097849903-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ae79401 at consistency null
2022-03-22 05:28:31.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1097849903-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:31.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [34289651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49210a87 at consistency null
2022-03-22 05:28:31.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [34289651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:31.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [469357174-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@218b6564 at consistency null
2022-03-22 05:28:31.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [469357174-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:32.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [733292221-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@632c85a9 at consistency null
2022-03-22 05:28:32.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [733292221-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:32.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344201536-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@555d4251 at consistency null
2022-03-22 05:28:32.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344201536-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:33.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340801164-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14e91f4f at consistency null
2022-03-22 05:28:33.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340801164-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:33.837 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:28:34.662 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:28:34.668 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-39, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:28:34.668 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-39, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:28:34.668 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-39, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:28:34.669 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-39, inFlight=0, closed=true] closing connection
2022-03-22 05:28:34.669 DEBUG 10 --- [uster-worker-67] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:28:34.670 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 05:28:34.670 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 32000 ms
2022-03-22 05:28:38.632 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-35, inFlight=0, closed=true] has already terminated
2022-03-22 05:28:38.866 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [450105701-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e33f8f3 at consistency null
2022-03-22 05:28:38.867 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [450105701-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:38.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [408308812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69b53951 at consistency null
2022-03-22 05:28:38.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [408308812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:39.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [741083167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56c5a1fc at consistency null
2022-03-22 05:28:39.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [741083167-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:39.162 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [995398262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a8acf0c at consistency null
2022-03-22 05:28:39.162 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [995398262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:39.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [20050752-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b8f4ace at consistency null
2022-03-22 05:28:39.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [20050752-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:39.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724560646-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bb08b5f at consistency null
2022-03-22 05:28:39.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724560646-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:39.979 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1737454220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40810794 at consistency null
2022-03-22 05:28:39.979 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1737454220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:40.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027391785-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1caf4227 at consistency null
2022-03-22 05:28:40.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027391785-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:40.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1995627648-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d77a246 at consistency null
2022-03-22 05:28:40.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1995627648-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:41.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [584150659-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a5736b3 at consistency null
2022-03-22 05:28:41.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [584150659-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:42.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1045852999-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c8fffa4 at consistency null
2022-03-22 05:28:42.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1045852999-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:42.469 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:28:47.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2048670055-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c4a09e1 at consistency null
2022-03-22 05:28:47.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2048670055-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:47.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [443214968-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f1de0b4 at consistency null
2022-03-22 05:28:47.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [443214968-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:47.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [870722752-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c2caf4a at consistency null
2022-03-22 05:28:47.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [870722752-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:47.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051180464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f2d1bb4 at consistency null
2022-03-22 05:28:47.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051180464-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:48.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [627796994-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43d408b1 at consistency null
2022-03-22 05:28:48.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [627796994-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:48.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1756319227-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@333c583f at consistency null
2022-03-22 05:28:48.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1756319227-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:49.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1789787138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ac3e7ed at consistency null
2022-03-22 05:28:49.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1789787138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:51.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776584410-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4de4afba at consistency null
2022-03-22 05:28:51.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776584410-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:51.128 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:28:51.245 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:28:51.406 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:28:56.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [395443352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e96a737 at consistency null
2022-03-22 05:28:56.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [395443352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:56.339 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96709380-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f3aff7a at consistency null
2022-03-22 05:28:56.339 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96709380-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:56.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [9116242-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ac383ff at consistency null
2022-03-22 05:28:56.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [9116242-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:56.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1293192911-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12b26a62 at consistency null
2022-03-22 05:28:56.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1293192911-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:56.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1442438857-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@986c81c at consistency null
2022-03-22 05:28:56.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1442438857-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:56.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1308280057-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21ef2779 at consistency null
2022-03-22 05:28:56.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1308280057-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:57.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [360386206-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@475594c0 at consistency null
2022-03-22 05:28:57.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [360386206-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:57.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2087753736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46b51732 at consistency null
2022-03-22 05:28:57.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2087753736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:57.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742999257-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@794ce1d0 at consistency null
2022-03-22 05:28:57.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742999257-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:58.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2082653136-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2140c56f at consistency null
2022-03-22 05:28:58.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2082653136-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:59.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [836775220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ad8a613 at consistency null
2022-03-22 05:28:59.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [836775220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:28:59.230 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:29:04.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235764184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a6603db at consistency null
2022-03-22 05:29:04.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235764184-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:04.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [251465458-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@124d07cd at consistency null
2022-03-22 05:29:04.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [251465458-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:04.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680568510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@672554b5 at consistency null
2022-03-22 05:29:04.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680568510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:04.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1418881154-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6dd7eb51 at consistency null
2022-03-22 05:29:04.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1418881154-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:04.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1605322571-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26243444 at consistency null
2022-03-22 05:29:04.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1605322571-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:04.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969103570-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c2d3c22 at consistency null
2022-03-22 05:29:04.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969103570-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:05.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [114450900-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11605bee at consistency null
2022-03-22 05:29:05.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [114450900-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:05.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1094963188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@433fb7f0 at consistency null
2022-03-22 05:29:05.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1094963188-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:05.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693666696-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@443e3dc7 at consistency null
2022-03-22 05:29:05.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693666696-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:05.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1196788748-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7147d54e at consistency null
2022-03-22 05:29:05.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1196788748-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:06.652 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1377602035-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62eb5fa3 at consistency null
2022-03-22 05:29:06.652 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1377602035-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:06.670 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:29:06.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-40, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:29:06.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-40, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:29:06.688 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-40, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:29:06.688 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-40, inFlight=0, closed=true] closing connection
2022-03-22 05:29:06.688 DEBUG 10 --- [uster-worker-69] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:29:06.689 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 05:29:06.689 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 64000 ms
2022-03-22 05:29:07.526 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [226480171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c55f439 at consistency null
2022-03-22 05:29:07.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [226480171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:07.528 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:29:12.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931363073-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69303a71 at consistency null
2022-03-22 05:29:12.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931363073-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:12.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [406881410-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26127fbc at consistency null
2022-03-22 05:29:12.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [406881410-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:12.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [782620008-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ebc3a9a at consistency null
2022-03-22 05:29:12.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [782620008-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:12.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571333663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d3bf148 at consistency null
2022-03-22 05:29:12.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571333663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:13.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555034336-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@530ad2c at consistency null
2022-03-22 05:29:13.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555034336-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:13.760 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079450467-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30eacda9 at consistency null
2022-03-22 05:29:13.760 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079450467-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:14.574 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079609492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f7d59c4 at consistency null
2022-03-22 05:29:14.574 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079609492-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:15.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1482245854-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d7e50d2 at consistency null
2022-03-22 05:29:15.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1482245854-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:16.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2077394373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d27db9c at consistency null
2022-03-22 05:29:16.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2077394373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:16.229 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:29:21.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [61212127-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b46c614 at consistency null
2022-03-22 05:29:21.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [61212127-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:21.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [255270935-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e500fc1 at consistency null
2022-03-22 05:29:21.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [255270935-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:21.407 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:29:21.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [491452569-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31a72019 at consistency null
2022-03-22 05:29:21.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [491452569-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:21.519 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:29:21.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1918478936-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@729ff68a at consistency null
2022-03-22 05:29:21.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1918478936-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:21.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096991025-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fc4e8ab at consistency null
2022-03-22 05:29:21.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096991025-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:22.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2011391634-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61d4ca95 at consistency null
2022-03-22 05:29:22.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2011391634-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:22.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026016967-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69fb895e at consistency null
2022-03-22 05:29:22.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026016967-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:22.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [34844753-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@645d194d at consistency null
2022-03-22 05:29:22.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [34844753-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:23.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1429176939-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ca424dd at consistency null
2022-03-22 05:29:23.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1429176939-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:24.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [957659562-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5be3c5be at consistency null
2022-03-22 05:29:24.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [957659562-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:24.709 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:29:29.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981881265-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e6716e5 at consistency null
2022-03-22 05:29:29.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981881265-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:29.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621342600-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@627ced15 at consistency null
2022-03-22 05:29:29.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621342600-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:29.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [211623320-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fc3af4b at consistency null
2022-03-22 05:29:29.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [211623320-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:30.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [282602568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ac85f38 at consistency null
2022-03-22 05:29:30.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [282602568-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:30.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [408677435-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ccea2ba at consistency null
2022-03-22 05:29:30.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [408677435-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:31.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2060268567-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@495ffa73 at consistency null
2022-03-22 05:29:31.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2060268567-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:32.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [509118219-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ba4ade4 at consistency null
2022-03-22 05:29:32.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [509118219-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:33.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1337152228-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@534d1237 at consistency null
2022-03-22 05:29:33.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1337152228-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:33.406 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:29:38.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139351325-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6aafb196 at consistency null
2022-03-22 05:29:38.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139351325-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:38.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [464561551-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72701f39 at consistency null
2022-03-22 05:29:38.467 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [464561551-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:38.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091180845-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@499777ea at consistency null
2022-03-22 05:29:38.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091180845-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:38.575 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [352016282-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ea34c4f at consistency null
2022-03-22 05:29:38.576 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [352016282-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:38.640 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860636325-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28d6168b at consistency null
2022-03-22 05:29:38.640 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860636325-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:38.679 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1894076971-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a2d4d7f at consistency null
2022-03-22 05:29:38.679 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1894076971-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:38.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1639444075-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@287749f7 at consistency null
2022-03-22 05:29:38.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1639444075-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:38.758 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [792605248-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c72b982 at consistency null
2022-03-22 05:29:38.758 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [792605248-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:38.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038072100-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@748888c9 at consistency null
2022-03-22 05:29:38.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038072100-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:38.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [592765482-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44fe7592 at consistency null
2022-03-22 05:29:38.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [592765482-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:39.175 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2110686365-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5654b40b at consistency null
2022-03-22 05:29:39.175 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2110686365-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:39.594 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [599524694-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f93b9ea at consistency null
2022-03-22 05:29:39.595 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [599524694-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:40.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250790310-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f6da211 at consistency null
2022-03-22 05:29:40.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250790310-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:40.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179517408-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2888068b at consistency null
2022-03-22 05:29:40.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179517408-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:41.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64008263-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b0f1b7d at consistency null
2022-03-22 05:29:41.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64008263-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:41.636 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:29:46.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082491870-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6961a51f at consistency null
2022-03-22 05:29:46.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082491870-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:46.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1491240345-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1beeff08 at consistency null
2022-03-22 05:29:46.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1491240345-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:46.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1266525522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@733a0ea0 at consistency null
2022-03-22 05:29:46.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1266525522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:46.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294563058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50cd3749 at consistency null
2022-03-22 05:29:46.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294563058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:47.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568236026-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34e03912 at consistency null
2022-03-22 05:29:47.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568236026-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:47.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1843138581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@581a8639 at consistency null
2022-03-22 05:29:47.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1843138581-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:47.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261268937-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b76cc94 at consistency null
2022-03-22 05:29:47.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261268937-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:48.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1303715512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42b8cbcc at consistency null
2022-03-22 05:29:48.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1303715512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:48.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402731443-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77255220 at consistency null
2022-03-22 05:29:48.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402731443-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:49.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [357615134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b8239b3 at consistency null
2022-03-22 05:29:49.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [357615134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:49.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [891527258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@365d583b at consistency null
2022-03-22 05:29:49.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [891527258-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:50.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1350969496-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32bf1397 at consistency null
2022-03-22 05:29:50.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1350969496-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:50.222 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:29:51.527 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:29:51.724 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:29:55.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1100679058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f56f3b2 at consistency null
2022-03-22 05:29:55.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1100679058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:55.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1390954889-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f3104a at consistency null
2022-03-22 05:29:55.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1390954889-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:55.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2053178110-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59b31c4f at consistency null
2022-03-22 05:29:55.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2053178110-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:55.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [465708994-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@135a9f45 at consistency null
2022-03-22 05:29:55.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [465708994-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:55.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [41686207-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b4925ff at consistency null
2022-03-22 05:29:55.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [41686207-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:56.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [306753912-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56251d9c at consistency null
2022-03-22 05:29:56.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [306753912-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:56.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1615603517-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17b52273 at consistency null
2022-03-22 05:29:56.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1615603517-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:57.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1790636225-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50522777 at consistency null
2022-03-22 05:29:57.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1790636225-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:58.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142727365-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5236865f at consistency null
2022-03-22 05:29:58.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142727365-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:59.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603627996-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c81a878 at consistency null
2022-03-22 05:29:59.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603627996-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:29:59.335 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:30:04.400 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1583467228-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1219a121 at consistency null
2022-03-22 05:30:04.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1583467228-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:04.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667237161-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5331458a at consistency null
2022-03-22 05:30:04.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667237161-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:04.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1696710256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@355325cd at consistency null
2022-03-22 05:30:04.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1696710256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:04.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [295629443-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14309977 at consistency null
2022-03-22 05:30:04.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [295629443-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:04.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1106898246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b7f2279 at consistency null
2022-03-22 05:30:04.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1106898246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:05.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [584311069-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@123b2389 at consistency null
2022-03-22 05:30:05.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [584311069-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:05.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1248397577-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e6a114e at consistency null
2022-03-22 05:30:05.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1248397577-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:05.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [746971181-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e01ccd5 at consistency null
2022-03-22 05:30:05.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [746971181-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:05.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [580434726-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cdf5f86 at consistency null
2022-03-22 05:30:05.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [580434726-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:06.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536767841-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@573310ff at consistency null
2022-03-22 05:30:06.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536767841-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:07.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [501030614-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39bd8b36 at consistency null
2022-03-22 05:30:07.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [501030614-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:07.112 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:30:10.689 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:30:10.693 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-41, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:30:10.693 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-41, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:30:10.693 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-41, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:30:10.695 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-41, inFlight=0, closed=true] closing connection
2022-03-22 05:30:10.696 DEBUG 10 --- [uster-worker-70] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:30:10.697 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 05:30:10.697 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 128000 ms
2022-03-22 05:30:12.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [517603784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@197a9fb0 at consistency null
2022-03-22 05:30:12.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [517603784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:12.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038220990-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fbfd076 at consistency null
2022-03-22 05:30:12.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038220990-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:12.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361047957-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30449396 at consistency null
2022-03-22 05:30:12.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361047957-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:12.262 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1625010269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65f55a93 at consistency null
2022-03-22 05:30:12.262 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1625010269-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:12.367 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [784415183-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28bfde82 at consistency null
2022-03-22 05:30:12.367 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [784415183-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:12.482 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [837238161-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29828783 at consistency null
2022-03-22 05:30:12.482 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [837238161-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:12.696 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [232918455-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dce7906 at consistency null
2022-03-22 05:30:12.699 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [232918455-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:13.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [814095630-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72a08dee at consistency null
2022-03-22 05:30:13.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [814095630-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:13.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397297040-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@243d16f6 at consistency null
2022-03-22 05:30:13.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397297040-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:15.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [531491137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19be336d at consistency null
2022-03-22 05:30:15.526 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [531491137-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:15.527 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:30:20.540 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46121142-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57767f15 at consistency null
2022-03-22 05:30:20.540 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46121142-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:20.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [38912898-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@214f57e8 at consistency null
2022-03-22 05:30:20.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [38912898-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:20.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1731630237-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d097f91 at consistency null
2022-03-22 05:30:20.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1731630237-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:20.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388079323-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c41c06 at consistency null
2022-03-22 05:30:20.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388079323-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:21.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801930489-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@612e457c at consistency null
2022-03-22 05:30:21.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801930489-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:21.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1730598340-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a85aea6 at consistency null
2022-03-22 05:30:21.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1730598340-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:21.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879179713-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cd6ab90 at consistency null
2022-03-22 05:30:21.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879179713-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:21.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [791392464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39d7b020 at consistency null
2022-03-22 05:30:21.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [791392464-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:21.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2116709186-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ac107c3 at consistency null
2022-03-22 05:30:21.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2116709186-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:21.727 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:30:21.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804460982-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d267ca3 at consistency null
2022-03-22 05:30:21.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804460982-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:21.912 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:30:22.143 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [581434038-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1485cabc at consistency null
2022-03-22 05:30:22.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [581434038-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:22.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1350071042-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10d73ebc at consistency null
2022-03-22 05:30:22.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1350071042-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:22.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2099601972-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@328acc7a at consistency null
2022-03-22 05:30:22.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2099601972-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:23.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1252846223-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cc76dec at consistency null
2022-03-22 05:30:23.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1252846223-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:23.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1464064806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ad80e2 at consistency null
2022-03-22 05:30:23.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1464064806-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:24.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1152402095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fa67c55 at consistency null
2022-03-22 05:30:24.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1152402095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:24.430 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:30:29.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [307408079-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b2b8cc7 at consistency null
2022-03-22 05:30:29.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [307408079-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:29.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065862296-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c042043 at consistency null
2022-03-22 05:30:29.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065862296-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:29.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1993650853-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26c403b1 at consistency null
2022-03-22 05:30:29.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1993650853-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:29.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808924138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a9c87e8 at consistency null
2022-03-22 05:30:29.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808924138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:30.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [931448311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2dc669c8 at consistency null
2022-03-22 05:30:30.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [931448311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:30.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [908485787-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@402d420b at consistency null
2022-03-22 05:30:30.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [908485787-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:30.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1697954144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46721834 at consistency null
2022-03-22 05:30:30.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1697954144-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:31.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821736527-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@188f64d5 at consistency null
2022-03-22 05:30:31.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821736527-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:31.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100274519-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@699d625 at consistency null
2022-03-22 05:30:31.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100274519-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:33.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2015892693-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c8ef1e7 at consistency null
2022-03-22 05:30:33.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2015892693-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:33.424 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:30:38.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941055241-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a6bfffa at consistency null
2022-03-22 05:30:38.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941055241-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:38.493 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2142710594-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@192195ae at consistency null
2022-03-22 05:30:38.494 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2142710594-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:38.598 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2142387454-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51c15051 at consistency null
2022-03-22 05:30:38.598 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2142387454-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:38.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1191444833-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a9b9247 at consistency null
2022-03-22 05:30:38.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1191444833-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:38.955 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1174004637-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f2a28c1 at consistency null
2022-03-22 05:30:38.955 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1174004637-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:39.167 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64109915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a33446f at consistency null
2022-03-22 05:30:39.167 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64109915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:39.380 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835643923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@612bc8a0 at consistency null
2022-03-22 05:30:39.380 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835643923-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:39.792 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [167886806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37f47768 at consistency null
2022-03-22 05:30:39.792 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [167886806-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:40.597 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [7229440-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49e63d77 at consistency null
2022-03-22 05:30:40.598 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [7229440-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:42.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [306486707-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69d61818 at consistency null
2022-03-22 05:30:42.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [306486707-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:42.206 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:30:47.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [850225860-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5607b34a at consistency null
2022-03-22 05:30:47.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [850225860-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:47.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [894798920-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42dcd1d5 at consistency null
2022-03-22 05:30:47.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [894798920-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:47.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1195975858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3aa9517c at consistency null
2022-03-22 05:30:47.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1195975858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:47.435 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1901494509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@636698e9 at consistency null
2022-03-22 05:30:47.435 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1901494509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:47.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [678702921-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@810fa2c at consistency null
2022-03-22 05:30:47.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [678702921-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:47.591 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [828035523-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ce6d082 at consistency null
2022-03-22 05:30:47.591 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [828035523-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:47.695 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1060638702-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cb456d3 at consistency null
2022-03-22 05:30:47.695 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1060638702-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:47.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049749260-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bc23c7c at consistency null
2022-03-22 05:30:47.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049749260-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:48.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1940641639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48c66b2 at consistency null
2022-03-22 05:30:48.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1940641639-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:49.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345560088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e74d461 at consistency null
2022-03-22 05:30:49.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345560088-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:51.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797320218-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60081bb2 at consistency null
2022-03-22 05:30:51.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797320218-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:51.013 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:30:51.919 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:30:52.030 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:30:56.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1956349603-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a33ea13 at consistency null
2022-03-22 05:30:56.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1956349603-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:56.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495689144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d6daa11 at consistency null
2022-03-22 05:30:56.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495689144-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:56.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [197935248-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2502b201 at consistency null
2022-03-22 05:30:56.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [197935248-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:56.455 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1735409613-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f7e898 at consistency null
2022-03-22 05:30:56.457 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1735409613-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:56.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341522718-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6071f947 at consistency null
2022-03-22 05:30:56.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341522718-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:56.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629651199-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10bcfa7a at consistency null
2022-03-22 05:30:56.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629651199-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:56.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1285995039-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@201ed0d6 at consistency null
2022-03-22 05:30:56.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1285995039-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:57.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718174737-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a174f65 at consistency null
2022-03-22 05:30:57.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718174737-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:57.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1440703141-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75c326a1 at consistency null
2022-03-22 05:30:57.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1440703141-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:57.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1308034735-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25b7d49 at consistency null
2022-03-22 05:30:57.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1308034735-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:57.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [880290636-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@541585b4 at consistency null
2022-03-22 05:30:57.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [880290636-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:58.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1832894752-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74599434 at consistency null
2022-03-22 05:30:58.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1832894752-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:58.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138445416-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18435230 at consistency null
2022-03-22 05:30:58.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138445416-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:59.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [913111234-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@412bd544 at consistency null
2022-03-22 05:30:59.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [913111234-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:30:59.335 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:31:04.433 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136367368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@396bd0f0 at consistency null
2022-03-22 05:31:04.433 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136367368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:04.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344174238-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ab0bdd at consistency null
2022-03-22 05:31:04.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344174238-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:04.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1228916473-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37b1b0a at consistency null
2022-03-22 05:31:04.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1228916473-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:04.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [415538676-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b6ec0f8 at consistency null
2022-03-22 05:31:04.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [415538676-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:05.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [131619379-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2242c9ab at consistency null
2022-03-22 05:31:05.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [131619379-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:05.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [614537438-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79ba25a2 at consistency null
2022-03-22 05:31:05.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [614537438-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:05.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [329128637-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e166e9e at consistency null
2022-03-22 05:31:05.531 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [329128637-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:06.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789122567-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43f20ea4 at consistency null
2022-03-22 05:31:06.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789122567-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:06.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1782960151-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77c988e0 at consistency null
2022-03-22 05:31:06.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1782960151-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:07.648 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175339625-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ff6348e at consistency null
2022-03-22 05:31:07.649 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175339625-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:07.651 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:31:12.665 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [631694311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2269c370 at consistency null
2022-03-22 05:31:12.665 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [631694311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:12.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1111009378-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71bd7eb7 at consistency null
2022-03-22 05:31:12.893 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1111009378-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:13.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [104406362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a8866e8 at consistency null
2022-03-22 05:31:13.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [104406362-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:13.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [848300798-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24347973 at consistency null
2022-03-22 05:31:13.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [848300798-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:13.456 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540769936-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48aa5cc6 at consistency null
2022-03-22 05:31:13.456 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540769936-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:13.871 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [659007821-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52f8dba2 at consistency null
2022-03-22 05:31:13.871 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [659007821-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:14.694 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1767547698-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b4e4833 at consistency null
2022-03-22 05:31:14.695 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1767547698-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:16.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1538233117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cb65dc8 at consistency null
2022-03-22 05:31:16.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1538233117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:16.312 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:435)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:31:21.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [538683654-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46e076f4 at consistency null
2022-03-22 05:31:21.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [538683654-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:21.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [982278537-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52519041 at consistency null
2022-03-22 05:31:21.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [982278537-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:21.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [108201689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b7755e8 at consistency null
2022-03-22 05:31:21.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [108201689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:21.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [560155249-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15c10f53 at consistency null
2022-03-22 05:31:21.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [560155249-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:21.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1556972738-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50d38cc9 at consistency null
2022-03-22 05:31:21.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1556972738-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:22.034 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:31:22.105 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:31:22.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [93599483-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e5ecd0a at consistency null
2022-03-22 05:31:22.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [93599483-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:22.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458210654-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fc7013c at consistency null
2022-03-22 05:31:22.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458210654-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:22.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541351044-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52612400 at consistency null
2022-03-22 05:31:22.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541351044-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:23.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270481370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e185f41 at consistency null
2022-03-22 05:31:23.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270481370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:23.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [882440396-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66d6e715 at consistency null
2022-03-22 05:31:23.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [882440396-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:23.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1390085708-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@116fc412 at consistency null
2022-03-22 05:31:23.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1390085708-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:24.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1358549231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a75a31b at consistency null
2022-03-22 05:31:24.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1358549231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:24.750 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:31:29.834 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [967937229-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16cb1f43 at consistency null
2022-03-22 05:31:29.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [967937229-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:29.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1251757487-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28d4bf78 at consistency null
2022-03-22 05:31:29.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1251757487-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:30.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090505686-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f5ebc02 at consistency null
2022-03-22 05:31:30.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090505686-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:30.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587479950-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12183ad at consistency null
2022-03-22 05:31:30.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587479950-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:30.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1811754356-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63512df5 at consistency null
2022-03-22 05:31:30.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1811754356-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:30.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134879889-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e737198 at consistency null
2022-03-22 05:31:30.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134879889-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:30.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2101028424-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54776ef8 at consistency null
2022-03-22 05:31:30.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2101028424-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:31.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1538195220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5946a330 at consistency null
2022-03-22 05:31:31.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1538195220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:32.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1505427352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33920368 at consistency null
2022-03-22 05:31:32.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1505427352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:33.045 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1037965394-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46984b33 at consistency null
2022-03-22 05:31:33.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1037965394-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:33.049 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:31:38.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [435380423-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29f7fdb6 at consistency null
2022-03-22 05:31:38.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [435380423-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:38.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889813682-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4544949b at consistency null
2022-03-22 05:31:38.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889813682-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:38.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1002505652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@666e5618 at consistency null
2022-03-22 05:31:38.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1002505652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:38.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989182998-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62ab5835 at consistency null
2022-03-22 05:31:38.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989182998-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:38.481 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1952920923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74a7139b at consistency null
2022-03-22 05:31:38.481 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1952920923-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:38.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180690319-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1526a1d8 at consistency null
2022-03-22 05:31:38.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180690319-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:38.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [966411718-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16cfbf2 at consistency null
2022-03-22 05:31:38.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [966411718-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:39.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1263064277-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39be611a at consistency null
2022-03-22 05:31:39.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1263064277-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:40.065 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [105446666-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7549867d at consistency null
2022-03-22 05:31:40.065 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [105446666-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:41.687 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [616800553-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55c1c16d at consistency null
2022-03-22 05:31:41.688 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [616800553-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:41.691 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:31:46.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1335649282-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3314034c at consistency null
2022-03-22 05:31:46.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1335649282-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:46.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1115680600-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79e81285 at consistency null
2022-03-22 05:31:46.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1115680600-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:46.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [52361904-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40ab19ab at consistency null
2022-03-22 05:31:46.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [52361904-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:47.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [988666131-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bba627a at consistency null
2022-03-22 05:31:47.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [988666131-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:47.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [377296440-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8a70c22 at consistency null
2022-03-22 05:31:47.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [377296440-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:47.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [725958272-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31624b89 at consistency null
2022-03-22 05:31:47.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [725958272-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:48.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1601866274-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@299c7a4f at consistency null
2022-03-22 05:31:48.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1601866274-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:49.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [185333151-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61fd64b4 at consistency null
2022-03-22 05:31:49.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [185333151-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:49.613 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:31:52.108 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:31:52.124 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:31:54.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1966734864-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@510ba2b5 at consistency null
2022-03-22 05:31:54.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1966734864-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:54.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2005565193-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7296851e at consistency null
2022-03-22 05:31:54.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2005565193-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:55.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2091565120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3812c701 at consistency null
2022-03-22 05:31:55.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2091565120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:55.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [685335879-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32c0cc73 at consistency null
2022-03-22 05:31:55.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [685335879-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:55.234 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [313182002-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e52ce94 at consistency null
2022-03-22 05:31:55.234 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [313182002-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:55.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [922517593-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@782a0235 at consistency null
2022-03-22 05:31:55.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [922517593-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:55.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [478080518-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@562ef8fe at consistency null
2022-03-22 05:31:55.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [478080518-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:56.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146321299-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6239151c at consistency null
2022-03-22 05:31:56.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146321299-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:56.640 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [993626123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ab751a1 at consistency null
2022-03-22 05:31:56.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [993626123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:57.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1762998653-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6224d968 at consistency null
2022-03-22 05:31:57.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1762998653-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:31:57.625 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:32:02.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49345671-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@436c34da at consistency null
2022-03-22 05:32:02.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49345671-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:02.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [252698001-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6957485e at consistency null
2022-03-22 05:32:02.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [252698001-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:02.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2021750187-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@191d268c at consistency null
2022-03-22 05:32:02.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2021750187-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:02.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941581591-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5db6535e at consistency null
2022-03-22 05:32:02.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941581591-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:03.045 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1036970591-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fde76d6 at consistency null
2022-03-22 05:32:03.045 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1036970591-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:03.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281121058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c18c4cb at consistency null
2022-03-22 05:32:03.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281121058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:03.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065571119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cc1ae82 at consistency null
2022-03-22 05:32:03.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065571119-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:04.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2096606741-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33023ae7 at consistency null
2022-03-22 05:32:04.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2096606741-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:04.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [908148678-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f669297 at consistency null
2022-03-22 05:32:04.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [908148678-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:05.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [403367787-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55b2372c at consistency null
2022-03-22 05:32:05.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [403367787-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:06.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1035326830-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d626a6 at consistency null
2022-03-22 05:32:06.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1035326830-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:06.135 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:32:11.148 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2087928779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2621f1d9 at consistency null
2022-03-22 05:32:11.148 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2087928779-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:11.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1125354636-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2dd82da0 at consistency null
2022-03-22 05:32:11.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1125354636-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:11.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208472634-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69977460 at consistency null
2022-03-22 05:32:11.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208472634-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:11.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [526430332-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cc18ce2 at consistency null
2022-03-22 05:32:11.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [526430332-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:11.672 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037105984-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7547019a at consistency null
2022-03-22 05:32:11.673 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037105984-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:12.090 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [650334477-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@191e87bf at consistency null
2022-03-22 05:32:12.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [650334477-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:12.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1117496117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bf8c488 at consistency null
2022-03-22 05:32:12.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1117496117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:13.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1482819658-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b8ee789 at consistency null
2022-03-22 05:32:13.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1482819658-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:14.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2131007427-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56087eb3 at consistency null
2022-03-22 05:32:14.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2131007427-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:14.915 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:32:18.697 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:32:18.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-42, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:32:18.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-42, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:32:18.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-42, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:32:18.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-42, inFlight=0, closed=true] closing connection
2022-03-22 05:32:18.718 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 256000 milliseconds
2022-03-22 05:32:18.718 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 256000 ms
2022-03-22 05:32:18.719 DEBUG 10 --- [uster-worker-71] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:32:19.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1140062743-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fa16c2e at consistency null
2022-03-22 05:32:19.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1140062743-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:20.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550744407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45a8dec0 at consistency null
2022-03-22 05:32:20.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550744407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:20.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344328392-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@480cb8e4 at consistency null
2022-03-22 05:32:20.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344328392-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:20.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340550932-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5935f5ab at consistency null
2022-03-22 05:32:20.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340550932-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:20.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [320358333-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d3122bf at consistency null
2022-03-22 05:32:20.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [320358333-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:20.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1015226954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1eecb107 at consistency null
2022-03-22 05:32:20.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1015226954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:20.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417840474-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@330ae1a6 at consistency null
2022-03-22 05:32:20.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417840474-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:20.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1846456029-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4564d2c1 at consistency null
2022-03-22 05:32:20.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1846456029-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:21.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1064970510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51543408 at consistency null
2022-03-22 05:32:21.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1064970510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:21.239 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [441372121-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a4bc3ac at consistency null
2022-03-22 05:32:21.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [441372121-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:21.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063698087-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e280092 at consistency null
2022-03-22 05:32:21.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063698087-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:21.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [604346320-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d39314c at consistency null
2022-03-22 05:32:21.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [604346320-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:22.143 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:32:22.226 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:32:22.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1255794695-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dbdb497 at consistency null
2022-03-22 05:32:22.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1255794695-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:22.818 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:32:27.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [638960768-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@291f4a2f at consistency null
2022-03-22 05:32:27.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [638960768-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:28.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [739034467-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27bde30e at consistency null
2022-03-22 05:32:28.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [739034467-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:28.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [608136229-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43ea05e4 at consistency null
2022-03-22 05:32:28.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [608136229-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:28.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [555954549-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73430802 at consistency null
2022-03-22 05:32:28.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [555954549-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:28.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [689618965-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@205e7a77 at consistency null
2022-03-22 05:32:28.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [689618965-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:28.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1668759432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@683348e1 at consistency null
2022-03-22 05:32:28.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1668759432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:28.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1014376839-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b65b526 at consistency null
2022-03-22 05:32:28.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1014376839-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:28.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82190309-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@100da150 at consistency null
2022-03-22 05:32:28.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82190309-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:28.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063822084-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4353ece4 at consistency null
2022-03-22 05:32:28.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063822084-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:28.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [559056531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3155c02f at consistency null
2022-03-22 05:32:28.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [559056531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:29.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [753248559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67cf9fa6 at consistency null
2022-03-22 05:32:29.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [753248559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:29.237 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [415797857-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ab5a52e at consistency null
2022-03-22 05:32:29.238 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [415797857-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:29.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2017699633-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46dd4e50 at consistency null
2022-03-22 05:32:29.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2017699633-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:30.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1301336244-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5996ba1c at consistency null
2022-03-22 05:32:30.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1301336244-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:31.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1287501048-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@423f3c27 at consistency null
2022-03-22 05:32:31.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1287501048-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:31.528 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:32:36.557 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [272268436-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75947d9b at consistency null
2022-03-22 05:32:36.557 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [272268436-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:36.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1965374349-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@591712b7 at consistency null
2022-03-22 05:32:36.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1965374349-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:36.669 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1048264941-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@353d05d3 at consistency null
2022-03-22 05:32:36.669 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1048264941-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:36.772 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532691374-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e58d957 at consistency null
2022-03-22 05:32:36.773 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532691374-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:36.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549613086-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58f615a2 at consistency null
2022-03-22 05:32:36.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549613086-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:36.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [7903563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@645bafdd at consistency null
2022-03-22 05:32:36.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [7903563-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:37.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1684069761-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c815060 at consistency null
2022-03-22 05:32:37.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1684069761-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:37.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054391274-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44320b57 at consistency null
2022-03-22 05:32:37.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054391274-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:37.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1331416946-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54020d4e at consistency null
2022-03-22 05:32:37.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1331416946-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:37.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [195933459-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d9e7554 at consistency null
2022-03-22 05:32:37.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [195933459-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:37.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1774447922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bba2ad7 at consistency null
2022-03-22 05:32:37.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1774447922-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:38.132 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [268007733-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f86f844 at consistency null
2022-03-22 05:32:38.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [268007733-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:38.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [508041191-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b2aee87 at consistency null
2022-03-22 05:32:38.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [508041191-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:38.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2016099344-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45801463 at consistency null
2022-03-22 05:32:38.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2016099344-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:39.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [242738266-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ff66296 at consistency null
2022-03-22 05:32:39.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [242738266-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:40.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1241715824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@779ae5f4 at consistency null
2022-03-22 05:32:40.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1241715824-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:40.039 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:32:45.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [346740216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b487356 at consistency null
2022-03-22 05:32:45.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [346740216-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:45.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [893432001-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e2e5350 at consistency null
2022-03-22 05:32:45.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [893432001-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:45.268 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [69519111-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52a064d1 at consistency null
2022-03-22 05:32:45.268 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [69519111-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:45.474 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1448534887-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72359e34 at consistency null
2022-03-22 05:32:45.474 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1448534887-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:45.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345110848-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4adf81e7 at consistency null
2022-03-22 05:32:45.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345110848-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:46.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619156907-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@627ef3f at consistency null
2022-03-22 05:32:46.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619156907-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:46.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [106525391-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f8c0d51 at consistency null
2022-03-22 05:32:46.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [106525391-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:46.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1744172216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6eb4388 at consistency null
2022-03-22 05:32:46.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1744172216-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:47.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2092260095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55e62826 at consistency null
2022-03-22 05:32:47.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2092260095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:48.237 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692914191-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f7f944d at consistency null
2022-03-22 05:32:48.237 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692914191-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:49.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [694491990-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50c6f676 at consistency null
2022-03-22 05:32:49.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [694491990-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:49.104 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:32:52.229 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:32:52.310 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:32:54.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644002038-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c6d08e5 at consistency null
2022-03-22 05:32:54.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644002038-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:54.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [927573652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@682eb656 at consistency null
2022-03-22 05:32:54.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [927573652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:54.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835302360-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b008ae2 at consistency null
2022-03-22 05:32:54.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835302360-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:54.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1826251913-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c12bab2 at consistency null
2022-03-22 05:32:54.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1826251913-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:55.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727695150-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@185cbff5 at consistency null
2022-03-22 05:32:55.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727695150-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:55.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486407779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55d26ae at consistency null
2022-03-22 05:32:55.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486407779-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:56.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1379519129-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a15fb7d at consistency null
2022-03-22 05:32:56.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1379519129-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:58.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836585595-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@733b05a7 at consistency null
2022-03-22 05:32:58.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836585595-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:32:58.031 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:33:03.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1601783043-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e65bf22 at consistency null
2022-03-22 05:33:03.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1601783043-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:03.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [746890857-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62619016 at consistency null
2022-03-22 05:33:03.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [746890857-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:03.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [813030498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@579a5134 at consistency null
2022-03-22 05:33:03.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [813030498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:03.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [498334685-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c27605d at consistency null
2022-03-22 05:33:03.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [498334685-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:03.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142079029-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f185c54 at consistency null
2022-03-22 05:33:03.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142079029-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:03.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [21609660-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31178681 at consistency null
2022-03-22 05:33:03.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [21609660-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:04.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [524412475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34016266 at consistency null
2022-03-22 05:33:04.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [524412475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:04.349 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497978141-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7363c933 at consistency null
2022-03-22 05:33:04.350 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497978141-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:04.575 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [709821121-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e84c1f7 at consistency null
2022-03-22 05:33:04.575 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [709821121-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:05.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1722808168-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@632a6128 at consistency null
2022-03-22 05:33:05.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1722808168-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:05.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1984935352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c073d79 at consistency null
2022-03-22 05:33:05.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1984935352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:05.829 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:33:10.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533741071-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31405d9b at consistency null
2022-03-22 05:33:10.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533741071-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:10.897 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1306230334-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7983b93d at consistency null
2022-03-22 05:33:10.897 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1306230334-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:10.960 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1423361114-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4910a404 at consistency null
2022-03-22 05:33:10.961 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1423361114-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:11.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1606371597-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c19b4aa at consistency null
2022-03-22 05:33:11.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1606371597-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:11.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098982579-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f068837 at consistency null
2022-03-22 05:33:11.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098982579-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:11.580 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1472038963-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ba890bb at consistency null
2022-03-22 05:33:11.580 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1472038963-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:11.984 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [892035722-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37543b4e at consistency null
2022-03-22 05:33:11.986 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [892035722-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:12.398 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276410883-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@568965ea at consistency null
2022-03-22 05:33:12.398 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276410883-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:12.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1486206979-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7de5183c at consistency null
2022-03-22 05:33:12.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1486206979-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:13.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674727218-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42777273 at consistency null
2022-03-22 05:33:13.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674727218-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:14.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [226709232-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@546faf38 at consistency null
2022-03-22 05:33:14.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [226709232-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:14.433 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:33:19.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1345785065-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49d4366a at consistency null
2022-03-22 05:33:19.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1345785065-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:19.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622326604-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ca5e394 at consistency null
2022-03-22 05:33:19.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622326604-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:19.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1414365955-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6538ee99 at consistency null
2022-03-22 05:33:19.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1414365955-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:19.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1135518644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c079b69 at consistency null
2022-03-22 05:33:19.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1135518644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:20.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [722906988-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@248893e5 at consistency null
2022-03-22 05:33:20.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [722906988-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:20.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [822195409-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17027da3 at consistency null
2022-03-22 05:33:20.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [822195409-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:20.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [166489193-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35ec05fc at consistency null
2022-03-22 05:33:20.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [166489193-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:20.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1072291299-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45400cfb at consistency null
2022-03-22 05:33:20.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1072291299-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:20.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [555378877-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25b7a80d at consistency null
2022-03-22 05:33:20.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [555378877-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:21.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486668147-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21dc367d at consistency null
2022-03-22 05:33:21.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486668147-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:21.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2011451294-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@744a145f at consistency null
2022-03-22 05:33:21.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2011451294-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:21.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [101945924-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3128e9f9 at consistency null
2022-03-22 05:33:21.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [101945924-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:22.086 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1958628795-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29c7d811 at consistency null
2022-03-22 05:33:22.086 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1958628795-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:22.311 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:33:22.325 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:33:22.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721140784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69686c8a at consistency null
2022-03-22 05:33:22.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721140784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:23.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [870034883-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4da47514 at consistency null
2022-03-22 05:33:23.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [870034883-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:23.106 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:33:26.141 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:33:26.147 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-57, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:33:26.147 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-57, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:33:26.148 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-57, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:33:26.149 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-57, inFlight=0, closed=true] closing connection
2022-03-22 05:33:26.149 DEBUG 10 --- [uster-worker-72] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:33:26.150 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 05:33:26.150 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 600000 ms
2022-03-22 05:33:28.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [700483662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bb52a78 at consistency null
2022-03-22 05:33:28.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [700483662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:28.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789957464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ca56a4f at consistency null
2022-03-22 05:33:28.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789957464-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:28.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103472012-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6034f6d at consistency null
2022-03-22 05:33:28.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103472012-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:28.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [331051094-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68e1966c at consistency null
2022-03-22 05:33:28.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [331051094-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:28.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1041504407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a1587e2 at consistency null
2022-03-22 05:33:28.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1041504407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:28.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1071270662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d657dfd at consistency null
2022-03-22 05:33:28.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1071270662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:29.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1094213502-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61012d1b at consistency null
2022-03-22 05:33:29.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1094213502-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:29.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155827919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b785a1 at consistency null
2022-03-22 05:33:29.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155827919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:30.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394780625-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53cb2862 at consistency null
2022-03-22 05:33:30.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394780625-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:31.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2062038108-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5172d73 at consistency null
2022-03-22 05:33:31.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2062038108-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:31.107 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:33:36.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2032687056-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d16a13f at consistency null
2022-03-22 05:33:36.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2032687056-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:36.187 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1028858364-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54b5abb1 at consistency null
2022-03-22 05:33:36.187 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1028858364-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:36.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344135888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62f7c36d at consistency null
2022-03-22 05:33:36.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344135888-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:36.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804422856-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ab08955 at consistency null
2022-03-22 05:33:36.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804422856-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:36.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [341819441-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43214591 at consistency null
2022-03-22 05:33:36.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [341819441-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:36.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [807886727-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@257b1861 at consistency null
2022-03-22 05:33:36.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [807886727-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:36.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013875915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@794b4b08 at consistency null
2022-03-22 05:33:36.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013875915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:37.143 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [117543754-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@470f9f55 at consistency null
2022-03-22 05:33:37.143 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [117543754-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:37.949 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1840616915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e4b47e8 at consistency null
2022-03-22 05:33:37.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1840616915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:39.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1734616197-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@738efe50 at consistency null
2022-03-22 05:33:39.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1734616197-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:39.572 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:33:44.579 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795131006-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50bef822 at consistency null
2022-03-22 05:33:44.579 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795131006-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:44.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [611639579-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bf02057 at consistency null
2022-03-22 05:33:44.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [611639579-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:44.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1705960267-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f505ec6 at consistency null
2022-03-22 05:33:44.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1705960267-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:44.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598254455-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ba84b02 at consistency null
2022-03-22 05:33:44.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598254455-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:44.753 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [690318384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cf0dbbd at consistency null
2022-03-22 05:33:44.753 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [690318384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:44.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [710552630-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76e8b2ec at consistency null
2022-03-22 05:33:44.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [710552630-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:45.068 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [956049037-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b736499 at consistency null
2022-03-22 05:33:45.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [956049037-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:45.489 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579780606-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32fad6ae at consistency null
2022-03-22 05:33:45.489 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579780606-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:45.900 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550789351-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@284dde7f at consistency null
2022-03-22 05:33:45.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550789351-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:46.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [665814720-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b817648 at consistency null
2022-03-22 05:33:46.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [665814720-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:47.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2126386422-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b869e0d at consistency null
2022-03-22 05:33:47.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2126386422-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:48.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1825178790-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fad6e75 at consistency null
2022-03-22 05:33:48.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1825178790-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:48.403 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:33:52.326 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:33:52.514 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:33:53.560 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1238836170-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6955e3d9 at consistency null
2022-03-22 05:33:53.561 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1238836170-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:53.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [311768502-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79f6fb8d at consistency null
2022-03-22 05:33:53.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [311768502-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:53.863 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [469932726-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67f7448 at consistency null
2022-03-22 05:33:53.863 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [469932726-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:53.937 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550584996-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42657c34 at consistency null
2022-03-22 05:33:53.937 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550584996-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:54.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1357446151-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@468148a at consistency null
2022-03-22 05:33:54.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1357446151-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:54.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1810586409-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a8cad19 at consistency null
2022-03-22 05:33:54.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1810586409-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:54.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1664400328-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@372c5d34 at consistency null
2022-03-22 05:33:54.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1664400328-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:54.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1230635368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ec11339 at consistency null
2022-03-22 05:33:54.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1230635368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:55.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1590019200-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a49802f at consistency null
2022-03-22 05:33:55.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1590019200-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:55.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1866923618-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3502121a at consistency null
2022-03-22 05:33:55.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1866923618-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:56.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703259321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60122c73 at consistency null
2022-03-22 05:33:56.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703259321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:57.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [807597997-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@369cb393 at consistency null
2022-03-22 05:33:57.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [807597997-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:33:57.127 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:34:02.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1077134573-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60020597 at consistency null
2022-03-22 05:34:02.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1077134573-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:02.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417240935-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45b5494f at consistency null
2022-03-22 05:34:02.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417240935-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:02.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1717518831-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34b5943e at consistency null
2022-03-22 05:34:02.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1717518831-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:02.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1097288982-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6766a910 at consistency null
2022-03-22 05:34:02.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1097288982-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:02.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [531492219-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17b836d6 at consistency null
2022-03-22 05:34:02.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [531492219-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:03.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931748422-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7828ea5f at consistency null
2022-03-22 05:34:03.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931748422-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:03.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1877496820-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60248500 at consistency null
2022-03-22 05:34:03.539 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1877496820-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:03.960 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [337423586-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32589e3d at consistency null
2022-03-22 05:34:03.960 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [337423586-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:04.799 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1688882479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64000026 at consistency null
2022-03-22 05:34:04.799 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1688882479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:04.801 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:34:09.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914244584-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bc574eb at consistency null
2022-03-22 05:34:09.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914244584-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:09.879 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [455393568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f0df616 at consistency null
2022-03-22 05:34:09.879 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [455393568-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:09.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1781400858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69d23abc at consistency null
2022-03-22 05:34:09.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1781400858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:09.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549532741-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d371690 at consistency null
2022-03-22 05:34:09.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549532741-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:10.044 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [419333925-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d2559f1 at consistency null
2022-03-22 05:34:10.044 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [419333925-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:10.160 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636381403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@751761ce at consistency null
2022-03-22 05:34:10.161 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636381403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:10.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1072873385-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40216a6a at consistency null
2022-03-22 05:34:10.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1072873385-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:10.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1012451268-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@179c594e at consistency null
2022-03-22 05:34:10.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1012451268-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:10.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349757125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f54de3d at consistency null
2022-03-22 05:34:10.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349757125-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:11.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109947552-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1103cd08 at consistency null
2022-03-22 05:34:11.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109947552-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:13.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [740835887-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a3a9778 at consistency null
2022-03-22 05:34:13.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [740835887-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:13.340 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:34:18.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1454687881-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f99f358 at consistency null
2022-03-22 05:34:18.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1454687881-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:18.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466381082-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@564565a0 at consistency null
2022-03-22 05:34:18.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466381082-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:18.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1895762025-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59a189ed at consistency null
2022-03-22 05:34:18.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1895762025-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:18.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [514327907-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@572e44f6 at consistency null
2022-03-22 05:34:18.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [514327907-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:18.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851263153-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37cd6e78 at consistency null
2022-03-22 05:34:18.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851263153-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:19.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371472601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ad86544 at consistency null
2022-03-22 05:34:19.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371472601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:19.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504419892-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fe36313 at consistency null
2022-03-22 05:34:19.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504419892-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:19.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1087196040-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a4f8ded at consistency null
2022-03-22 05:34:19.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1087196040-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:19.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1853808288-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66ad9a72 at consistency null
2022-03-22 05:34:19.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1853808288-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:20.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1496043846-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fd621a6 at consistency null
2022-03-22 05:34:20.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1496043846-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:22.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091702049-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ba93cb4 at consistency null
2022-03-22 05:34:22.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091702049-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:22.247 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:34:22.515 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:34:22.527 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:34:27.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806274432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2dcff4c9 at consistency null
2022-03-22 05:34:27.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806274432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:27.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1933232011-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27bb8508 at consistency null
2022-03-22 05:34:27.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1933232011-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:27.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1956419915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@793e4b48 at consistency null
2022-03-22 05:34:27.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1956419915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:27.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632748540-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b509447 at consistency null
2022-03-22 05:34:27.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632748540-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:27.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46446405-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@96b4a58 at consistency null
2022-03-22 05:34:27.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46446405-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:27.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [669462025-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72031db8 at consistency null
2022-03-22 05:34:27.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [669462025-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:27.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [933346687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79ec6cbb at consistency null
2022-03-22 05:34:27.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [933346687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:28.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [634051438-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26c13d6d at consistency null
2022-03-22 05:34:28.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [634051438-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:28.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43204109-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75ab9a67 at consistency null
2022-03-22 05:34:28.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43204109-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:29.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [471519024-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fafb7b7 at consistency null
2022-03-22 05:34:29.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [471519024-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:31.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [644531081-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53c894d at consistency null
2022-03-22 05:34:31.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [644531081-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:31.208 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:34:36.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1426619777-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ab9f6d2 at consistency null
2022-03-22 05:34:36.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1426619777-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:36.279 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324368507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3683940 at consistency null
2022-03-22 05:34:36.279 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324368507-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:36.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72972250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61182495 at consistency null
2022-03-22 05:34:36.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72972250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:36.444 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1952418387-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65e5c6bf at consistency null
2022-03-22 05:34:36.444 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1952418387-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:36.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1655510316-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2705417a at consistency null
2022-03-22 05:34:36.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1655510316-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:36.658 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [771907736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@799e583 at consistency null
2022-03-22 05:34:36.658 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [771907736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:36.767 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630151973-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a13fd05 at consistency null
2022-03-22 05:34:36.767 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630151973-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:36.873 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1773086273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31080cc6 at consistency null
2022-03-22 05:34:36.873 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1773086273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:36.985 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1671518765-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c8b950c at consistency null
2022-03-22 05:34:36.985 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1671518765-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:37.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587212385-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74f6cae3 at consistency null
2022-03-22 05:34:37.092 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587212385-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:37.300 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1848243610-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1168eaad at consistency null
2022-03-22 05:34:37.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1848243610-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:37.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2124700464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6077ba7a at consistency null
2022-03-22 05:34:37.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2124700464-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:37.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [398264059-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b4608d at consistency null
2022-03-22 05:34:37.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [398264059-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:38.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [506328469-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42d703aa at consistency null
2022-03-22 05:34:38.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [506328469-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:39.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1057576687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14bcb4df at consistency null
2022-03-22 05:34:39.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1057576687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:39.531 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:34:44.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1672013797-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49c57858 at consistency null
2022-03-22 05:34:44.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1672013797-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:44.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475905973-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a56d9e5 at consistency null
2022-03-22 05:34:44.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475905973-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:44.597 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1840079094-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10665d56 at consistency null
2022-03-22 05:34:44.597 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1840079094-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:44.653 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417791362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@618d09c9 at consistency null
2022-03-22 05:34:44.653 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417791362-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:44.759 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1012250292-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@368aa18 at consistency null
2022-03-22 05:34:44.759 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1012250292-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:44.865 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1292113564-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fadc261 at consistency null
2022-03-22 05:34:44.865 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1292113564-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:44.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1269327130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d03a82f at consistency null
2022-03-22 05:34:44.987 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1269327130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:45.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1678609152-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74492507 at consistency null
2022-03-22 05:34:45.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1678609152-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:45.299 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1979474211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@323769fc at consistency null
2022-03-22 05:34:45.299 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1979474211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:45.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [404830710-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b80b11 at consistency null
2022-03-22 05:34:45.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [404830710-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:46.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [279519069-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@639a8999 at consistency null
2022-03-22 05:34:46.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [279519069-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:46.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [410309571-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e46e9f0 at consistency null
2022-03-22 05:34:46.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [410309571-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:48.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [244800663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5737134d at consistency null
2022-03-22 05:34:48.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [244800663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:48.605 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:34:52.545 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:34:52.552 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:34:53.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [535312954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14947a39 at consistency null
2022-03-22 05:34:53.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [535312954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:53.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197706605-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2375ef73 at consistency null
2022-03-22 05:34:53.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197706605-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:53.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2033689737-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a158b84 at consistency null
2022-03-22 05:34:53.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2033689737-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:53.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1307585756-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@255754f0 at consistency null
2022-03-22 05:34:53.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1307585756-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:54.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [320337586-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12a5b7d1 at consistency null
2022-03-22 05:34:54.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [320337586-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:54.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1212418392-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e27cc81 at consistency null
2022-03-22 05:34:54.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1212418392-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:54.159 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [202633013-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78883d6a at consistency null
2022-03-22 05:34:54.159 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [202633013-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:54.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1078568103-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6767165f at consistency null
2022-03-22 05:34:54.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1078568103-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:54.335 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1825007738-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58433e5d at consistency null
2022-03-22 05:34:54.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1825007738-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:54.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [69313139-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78a0b210 at consistency null
2022-03-22 05:34:54.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [69313139-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:54.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391683973-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41faa303 at consistency null
2022-03-22 05:34:54.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391683973-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:55.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [439975253-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e70808d at consistency null
2022-03-22 05:34:55.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [439975253-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:55.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1307202256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f80f4c8 at consistency null
2022-03-22 05:34:55.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1307202256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:55.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1146176292-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@224ded1a at consistency null
2022-03-22 05:34:55.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1146176292-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:56.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396777628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36bbb0eb at consistency null
2022-03-22 05:34:56.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396777628-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:57.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176788200-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cd52a59 at consistency null
2022-03-22 05:34:57.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176788200-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:34:57.021 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:35:02.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506277934-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@742bbdbf at consistency null
2022-03-22 05:35:02.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506277934-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:02.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390187601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3943bb1d at consistency null
2022-03-22 05:35:02.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390187601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:02.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839258781-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52947836 at consistency null
2022-03-22 05:35:02.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839258781-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:02.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1453443881-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@579617df at consistency null
2022-03-22 05:35:02.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1453443881-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:02.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [810638681-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ef98ed7 at consistency null
2022-03-22 05:35:02.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [810638681-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:02.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [693836352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ac64946 at consistency null
2022-03-22 05:35:02.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [693836352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:03.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [734750476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b62cfe0 at consistency null
2022-03-22 05:35:03.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [734750476-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:05.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936394719-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c2d0dbc at consistency null
2022-03-22 05:35:05.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936394719-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:05.354 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:35:10.360 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1240086910-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e10bdd8 at consistency null
2022-03-22 05:35:10.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1240086910-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:10.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [940749593-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16745725 at consistency null
2022-03-22 05:35:10.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [940749593-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:10.474 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1209094160-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b5487c4 at consistency null
2022-03-22 05:35:10.475 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1209094160-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:10.584 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1671400764-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f302fff at consistency null
2022-03-22 05:35:10.584 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1671400764-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:10.791 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066119160-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74b5adbc at consistency null
2022-03-22 05:35:10.791 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066119160-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:11.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1451329534-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40cc1170 at consistency null
2022-03-22 05:35:11.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1451329534-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:11.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [349359342-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f29c40a at consistency null
2022-03-22 05:35:11.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [349359342-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:11.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [188433701-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41e855c7 at consistency null
2022-03-22 05:35:11.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [188433701-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:12.060 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134167043-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18e9d80b at consistency null
2022-03-22 05:35:12.060 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134167043-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:12.476 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [427742713-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@380e0df7 at consistency null
2022-03-22 05:35:12.476 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [427742713-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:13.281 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466210692-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39068768 at consistency null
2022-03-22 05:35:13.281 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466210692-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:14.097 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1250599861-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ac3bbe2 at consistency null
2022-03-22 05:35:14.097 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1250599861-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:14.099 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:35:19.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495473403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cbc1531 at consistency null
2022-03-22 05:35:19.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495473403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:19.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916310843-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2415cce4 at consistency null
2022-03-22 05:35:19.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916310843-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:19.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1677559683-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29a522c9 at consistency null
2022-03-22 05:35:19.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1677559683-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:19.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2096437058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e9e90b4 at consistency null
2022-03-22 05:35:19.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2096437058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:19.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1039141091-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20719cb at consistency null
2022-03-22 05:35:19.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1039141091-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:20.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1701632117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@87f9783 at consistency null
2022-03-22 05:35:20.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1701632117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:20.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1780648644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8cc5f2f at consistency null
2022-03-22 05:35:20.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1780648644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:21.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1043980229-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@722d65b2 at consistency null
2022-03-22 05:35:21.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1043980229-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:21.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [53702641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@485ee37d at consistency null
2022-03-22 05:35:21.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [53702641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:22.571 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:35:22.609 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:35:22.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [10465426-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ce9b67a at consistency null
2022-03-22 05:35:22.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [10465426-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:22.804 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:35:27.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2115333871-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5636b3ef at consistency null
2022-03-22 05:35:27.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2115333871-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:27.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245514527-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f2f4c6e at consistency null
2022-03-22 05:35:27.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245514527-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:28.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [376896477-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70550bfd at consistency null
2022-03-22 05:35:28.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [376896477-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:28.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1804649121-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fe39eb at consistency null
2022-03-22 05:35:28.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1804649121-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:28.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051175722-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62b44bf0 at consistency null
2022-03-22 05:35:28.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051175722-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:28.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579972952-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57cd37f4 at consistency null
2022-03-22 05:35:28.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579972952-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:29.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60297412-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f1fd413 at consistency null
2022-03-22 05:35:29.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60297412-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:29.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1852854236-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f5e957a at consistency null
2022-03-22 05:35:29.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1852854236-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:29.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1829590979-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f75aaaa at consistency null
2022-03-22 05:35:29.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1829590979-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:29.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112137301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1394827 at consistency null
2022-03-22 05:35:29.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112137301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:30.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18147560-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70f7bb5f at consistency null
2022-03-22 05:35:30.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18147560-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:30.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836472761-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b09057e at consistency null
2022-03-22 05:35:30.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836472761-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:30.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1791146139-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69bbd04f at consistency null
2022-03-22 05:35:30.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1791146139-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:31.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727359200-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2208c52f at consistency null
2022-03-22 05:35:31.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727359200-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:31.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082171262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fe8af72 at consistency null
2022-03-22 05:35:31.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082171262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:31.724 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:35:36.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [526695750-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cf96504 at consistency null
2022-03-22 05:35:36.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [526695750-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:36.766 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620957265-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bf7ef7c at consistency null
2022-03-22 05:35:36.766 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620957265-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:36.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769793738-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fb26e90 at consistency null
2022-03-22 05:35:36.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769793738-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:36.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776095079-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@258bd32b at consistency null
2022-03-22 05:35:36.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776095079-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:37.076 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399390615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@324d43db at consistency null
2022-03-22 05:35:37.076 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399390615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:37.192 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1563952379-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6695c6e1 at consistency null
2022-03-22 05:35:37.193 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1563952379-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:37.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611748824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a4fe24d at consistency null
2022-03-22 05:35:37.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611748824-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:37.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630980234-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12674a4 at consistency null
2022-03-22 05:35:37.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630980234-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:38.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2051541122-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@592dbcc0 at consistency null
2022-03-22 05:35:38.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2051541122-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:38.458 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741346028-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1147dbca at consistency null
2022-03-22 05:35:38.458 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741346028-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:39.272 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [228586072-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@141f1e25 at consistency null
2022-03-22 05:35:39.273 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [228586072-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:39.275 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:35:44.288 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1522141689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26aedb56 at consistency null
2022-03-22 05:35:44.288 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1522141689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:44.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611400756-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1251b8ff at consistency null
2022-03-22 05:35:44.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611400756-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:44.475 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1364251501-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d67b8f4 at consistency null
2022-03-22 05:35:44.475 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1364251501-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:44.685 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [590341677-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51e5674b at consistency null
2022-03-22 05:35:44.685 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [590341677-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:44.889 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [701552240-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@141724a at consistency null
2022-03-22 05:35:44.889 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [701552240-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:45.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703416536-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ec8b4dd at consistency null
2022-03-22 05:35:45.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703416536-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:45.499 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2129739059-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e681fe8 at consistency null
2022-03-22 05:35:45.499 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2129739059-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:45.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [582201934-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64cdec51 at consistency null
2022-03-22 05:35:45.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [582201934-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:46.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [226464574-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e9401a0 at consistency null
2022-03-22 05:35:46.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [226464574-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:46.718 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:35:51.777 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318564289-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35213ef3 at consistency null
2022-03-22 05:35:51.777 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318564289-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:51.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [972555653-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b68e0e0 at consistency null
2022-03-22 05:35:51.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [972555653-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:51.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [115794230-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@604c26b8 at consistency null
2022-03-22 05:35:51.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [115794230-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:52.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012308663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e1ba210 at consistency null
2022-03-22 05:35:52.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012308663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:52.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68934193-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1271d786 at consistency null
2022-03-22 05:35:52.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68934193-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:52.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100699930-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e324ac8 at consistency null
2022-03-22 05:35:52.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100699930-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:52.610 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:35:52.616 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:35:52.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1290306267-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fcac250 at consistency null
2022-03-22 05:35:52.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1290306267-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:53.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [809538349-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c32ec9b at consistency null
2022-03-22 05:35:53.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [809538349-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:53.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1783635270-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50e3df7c at consistency null
2022-03-22 05:35:53.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1783635270-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:53.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [640863717-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cababa2 at consistency null
2022-03-22 05:35:53.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [640863717-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:54.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776236450-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@151a2e25 at consistency null
2022-03-22 05:35:54.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776236450-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:55.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [438917820-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e175e29 at consistency null
2022-03-22 05:35:55.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [438917820-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:35:55.634 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:36:00.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887030930-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c160a8 at consistency null
2022-03-22 05:36:00.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887030930-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:00.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083467933-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65b954b8 at consistency null
2022-03-22 05:36:00.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083467933-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:00.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1295004752-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@112bb830 at consistency null
2022-03-22 05:36:00.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1295004752-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:01.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [496790512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ae44e9d at consistency null
2022-03-22 05:36:01.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [496790512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:01.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598682511-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a0fc30d at consistency null
2022-03-22 05:36:01.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598682511-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:01.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083994087-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@253c5a60 at consistency null
2022-03-22 05:36:01.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083994087-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:01.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [205668624-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@691a9ccf at consistency null
2022-03-22 05:36:01.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [205668624-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:01.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1929341059-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5132a7d7 at consistency null
2022-03-22 05:36:01.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1929341059-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:02.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [685164032-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25d10157 at consistency null
2022-03-22 05:36:02.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [685164032-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:02.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [194406314-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6279b0d7 at consistency null
2022-03-22 05:36:02.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [194406314-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:03.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365096201-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f38a4b3 at consistency null
2022-03-22 05:36:03.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365096201-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:03.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [485499970-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b70ac9c at consistency null
2022-03-22 05:36:03.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [485499970-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:04.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [34084634-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3606dd1c at consistency null
2022-03-22 05:36:04.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [34084634-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:04.220 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:36:09.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [974887878-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19562c03 at consistency null
2022-03-22 05:36:09.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [974887878-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:09.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [951292713-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3511ea6 at consistency null
2022-03-22 05:36:09.297 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [951292713-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:09.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [628912713-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d5b561c at consistency null
2022-03-22 05:36:09.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [628912713-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:09.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728971067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1db59696 at consistency null
2022-03-22 05:36:09.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728971067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:09.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [19755959-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35ab8717 at consistency null
2022-03-22 05:36:09.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [19755959-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:09.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [631472736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e041b2a at consistency null
2022-03-22 05:36:09.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [631472736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:10.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1670070951-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79a85140 at consistency null
2022-03-22 05:36:10.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1670070951-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:11.162 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1947963444-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@489869c5 at consistency null
2022-03-22 05:36:11.162 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1947963444-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:12.784 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [516027851-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21e2dcfe at consistency null
2022-03-22 05:36:12.784 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [516027851-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:12.785 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:36:17.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [667473137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fa14ea3 at consistency null
2022-03-22 05:36:17.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [667473137-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:17.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510604501-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4382a46 at consistency null
2022-03-22 05:36:17.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510604501-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:18.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [920493340-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fe5bd45 at consistency null
2022-03-22 05:36:18.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [920493340-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:18.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [656628129-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38d7956b at consistency null
2022-03-22 05:36:18.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [656628129-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:18.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1177386728-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@db61084 at consistency null
2022-03-22 05:36:18.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1177386728-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:18.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [562246500-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4819716e at consistency null
2022-03-22 05:36:18.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [562246500-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:19.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2070703925-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a34e2bd at consistency null
2022-03-22 05:36:19.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2070703925-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:19.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1178591144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ecc5e5d at consistency null
2022-03-22 05:36:19.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1178591144-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:20.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1799889849-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53c34130 at consistency null
2022-03-22 05:36:20.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1799889849-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:21.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407722919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50d2c827 at consistency null
2022-03-22 05:36:21.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407722919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:21.103 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:36:22.638 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:36:22.713 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:36:26.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273359656-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d3ee98a at consistency null
2022-03-22 05:36:26.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273359656-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:26.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931454819-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@118c72c6 at consistency null
2022-03-22 05:36:26.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931454819-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:26.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499466066-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@588f7b06 at consistency null
2022-03-22 05:36:26.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499466066-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:26.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261947588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1eda03c0 at consistency null
2022-03-22 05:36:26.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261947588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:26.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [794812967-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74c2cb28 at consistency null
2022-03-22 05:36:26.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [794812967-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:26.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [553980446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6aa9d34 at consistency null
2022-03-22 05:36:26.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [553980446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:26.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029434357-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@255e81ac at consistency null
2022-03-22 05:36:26.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029434357-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:27.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [224539077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b5452c4 at consistency null
2022-03-22 05:36:27.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [224539077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:27.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1568453780-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9ad4f0d at consistency null
2022-03-22 05:36:27.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1568453780-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:27.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415633095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bbc3031 at consistency null
2022-03-22 05:36:27.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415633095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:27.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [435459747-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@abe890b at consistency null
2022-03-22 05:36:27.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [435459747-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:28.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [92565569-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43b538ea at consistency null
2022-03-22 05:36:28.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [92565569-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:28.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1030012017-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2003d2b6 at consistency null
2022-03-22 05:36:28.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1030012017-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:28.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871773178-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a84433f at consistency null
2022-03-22 05:36:28.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871773178-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:29.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1391214373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@648c67b5 at consistency null
2022-03-22 05:36:29.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1391214373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:29.720 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:36:34.719 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:36:34.730 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-43, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:36:34.731 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-43, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:36:34.731 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-43, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:36:34.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959545665-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e9461a6 at consistency null
2022-03-22 05:36:34.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959545665-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:34.731 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-43, inFlight=0, closed=true] closing connection
2022-03-22 05:36:34.732 DEBUG 10 --- [uster-worker-73] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:36:34.733 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 512000 milliseconds
2022-03-22 05:36:34.733 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 512000 ms
2022-03-22 05:36:34.760 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1817441949-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30bc84a0 at consistency null
2022-03-22 05:36:34.760 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1817441949-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:34.798 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [79997376-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@819a433 at consistency null
2022-03-22 05:36:34.798 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [79997376-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:34.858 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [937605994-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f2fdb4a at consistency null
2022-03-22 05:36:34.858 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [937605994-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:34.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [560101524-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@229b0258 at consistency null
2022-03-22 05:36:34.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [560101524-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:34.979 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383503163-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76d60b2f at consistency null
2022-03-22 05:36:34.980 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383503163-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:35.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1935522844-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@332b01fa at consistency null
2022-03-22 05:36:35.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1935522844-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:35.094 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1371959542-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@422f695e at consistency null
2022-03-22 05:36:35.094 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1371959542-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:35.164 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674708098-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e95d1c6 at consistency null
2022-03-22 05:36:35.164 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674708098-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:35.271 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [99322510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@155a3b3f at consistency null
2022-03-22 05:36:35.272 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [99322510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:35.476 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [457730931-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c06c553 at consistency null
2022-03-22 05:36:35.476 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [457730931-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:35.681 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037115180-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75576a70 at consistency null
2022-03-22 05:36:35.681 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037115180-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:36.085 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [183370903-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@251c8efd at consistency null
2022-03-22 05:36:36.085 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [183370903-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:36.893 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1393049790-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b6da787 at consistency null
2022-03-22 05:36:36.893 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1393049790-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:38.499 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [188983903-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35274670 at consistency null
2022-03-22 05:36:38.499 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [188983903-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:38.501 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:36:43.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [759627922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@740d4d4f at consistency null
2022-03-22 05:36:43.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [759627922-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.539 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475834641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b3c07a6 at consistency null
2022-03-22 05:36:43.539 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475834641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.568 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1189856481-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d7bd0d9 at consistency null
2022-03-22 05:36:43.568 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1189856481-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1357680433-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@526019c0 at consistency null
2022-03-22 05:36:43.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1357680433-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1169839662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c1951f6 at consistency null
2022-03-22 05:36:43.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1169839662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.677 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [691653235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67337ee4 at consistency null
2022-03-22 05:36:43.677 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [691653235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [837672825-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64cd87c3 at consistency null
2022-03-22 05:36:43.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [837672825-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [88890226-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36b12c83 at consistency null
2022-03-22 05:36:43.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [88890226-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.784 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315647699-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30215d42 at consistency null
2022-03-22 05:36:43.785 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315647699-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1977441448-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c99c1fc at consistency null
2022-03-22 05:36:43.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1977441448-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [705975165-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44e2a433 at consistency null
2022-03-22 05:36:43.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [705975165-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298406022-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b885d18 at consistency null
2022-03-22 05:36:43.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298406022-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1544751879-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4179d466 at consistency null
2022-03-22 05:36:43.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1544751879-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:43.979 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1135883313-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@436d6b94 at consistency null
2022-03-22 05:36:43.979 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1135883313-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:44.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261635713-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@127f8cf6 at consistency null
2022-03-22 05:36:44.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261635713-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:44.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [653171531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b26733c at consistency null
2022-03-22 05:36:44.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [653171531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:44.360 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1658972066-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@247348da at consistency null
2022-03-22 05:36:44.360 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1658972066-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:44.767 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1458803-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ceaa60b at consistency null
2022-03-22 05:36:44.767 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1458803-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:45.571 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1404184048-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@621c835c at consistency null
2022-03-22 05:36:45.571 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1404184048-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:47.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [30726158-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7abbb3b0 at consistency null
2022-03-22 05:36:47.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [30726158-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:47.206 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:36:52.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1818620769-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44b001a5 at consistency null
2022-03-22 05:36:52.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1818620769-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:52.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1582440772-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b334b5f at consistency null
2022-03-22 05:36:52.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1582440772-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:52.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [470618843-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c324dba at consistency null
2022-03-22 05:36:52.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [470618843-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:52.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1002356220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78cbceb0 at consistency null
2022-03-22 05:36:52.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1002356220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:52.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2013511441-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c120162 at consistency null
2022-03-22 05:36:52.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2013511441-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:52.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [932505940-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d7ac40c at consistency null
2022-03-22 05:36:52.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [932505940-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:52.714 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:36:52.718 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:36:52.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1140567735-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@530b1984 at consistency null
2022-03-22 05:36:52.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1140567735-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:53.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [224701033-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7513f31b at consistency null
2022-03-22 05:36:53.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [224701033-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:53.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1761245155-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1966f5ca at consistency null
2022-03-22 05:36:53.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1761245155-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:53.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2016738785-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3217084e at consistency null
2022-03-22 05:36:53.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2016738785-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:53.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1713666630-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@395bf0b1 at consistency null
2022-03-22 05:36:53.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1713666630-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:54.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1320919817-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@533a47aa at consistency null
2022-03-22 05:36:54.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1320919817-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:55.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1975881810-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ddc20a2 at consistency null
2022-03-22 05:36:55.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1975881810-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:56.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [708944126-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7703022f at consistency null
2022-03-22 05:36:56.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [708944126-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:36:56.010 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:37:01.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1878133178-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11bbba06 at consistency null
2022-03-22 05:37:01.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1878133178-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:01.051 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2084723878-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@103db88c at consistency null
2022-03-22 05:37:01.051 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2084723878-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:01.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [635176060-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c785319 at consistency null
2022-03-22 05:37:01.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [635176060-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:01.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [37557865-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63852ac3 at consistency null
2022-03-22 05:37:01.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [37557865-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:01.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2113757261-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fa2c73f at consistency null
2022-03-22 05:37:01.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2113757261-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:01.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [330609720-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52a6e39f at consistency null
2022-03-22 05:37:01.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [330609720-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:01.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1452191190-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c088249 at consistency null
2022-03-22 05:37:01.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1452191190-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:01.649 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013606192-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f33ceeb at consistency null
2022-03-22 05:37:01.649 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013606192-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:01.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [713263096-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52aea347 at consistency null
2022-03-22 05:37:01.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [713263096-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:02.264 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [471921636-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fe31193 at consistency null
2022-03-22 05:37:02.264 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [471921636-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:03.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063456171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ed6d212 at consistency null
2022-03-22 05:37:03.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063456171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:04.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1356836496-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17470f7b at consistency null
2022-03-22 05:37:04.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1356836496-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:37:04.693 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:27:54.325Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:37:22.731 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:37:22.806 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:37:33.918 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:31:40Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:37:43.952 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:31:40Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:37:52.807 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:37:52.837 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:38:02.110 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UP /10.233.66.75:9042, scheduling delivery
2022-03-22 05:38:02.110 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event UP
2022-03-22 05:38:03.112 DEBUG 10 --- [uster-worker-74] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.233.66.75:9042
2022-03-22 05:38:03.215 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host UP
2022-03-22 05:38:03.216 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2022-03-22 05:38:03.216 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.233.66.75:9042
2022-03-22 05:38:03.216 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:38:03.231 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-58, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:38:07.708 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:38:17.736 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:38:19.315 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-58, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 05:38:19.315 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-58, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:38:19.316 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-58, inFlight=0, closed=true] closing connection
2022-03-22 05:38:19.317 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-58, inFlight=0, closed=true] has already terminated
2022-03-22 05:38:19.317 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:38:19.330 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-59, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:38:26.421 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-59, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 05:38:26.421 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-59, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:38:26.421 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-59, inFlight=0, closed=true] closing connection
2022-03-22 05:38:26.423 WARN 10 --- [er-nio-worker-2] com.datastax.driver.core.Session : Error creating pool to /10.233.66.75:9042
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 05:38:26.424 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-59, inFlight=0, closed=true] has already terminated
2022-03-22 05:38:26.429 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Cluster : Connection pool cannot be created, not marking /10.233.66.75:9042 UP
2022-03-22 05:38:26.430 DEBUG 10 --- [uster-worker-75] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] marking host DOWN
2022-03-22 05:38:26.430 DEBUG 10 --- [uster-worker-75] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2022-03-22 05:38:26.430 DEBUG 10 --- [uster-worker-75] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] starting reconnection attempt
2022-03-22 05:38:26.431 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:38:27.431 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:38:27.462 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-60, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:38:27.816 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:38:32.934 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-60, inFlight=0, closed=false] because: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 05:38:32.935 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-60, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:38:32.935 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-60, inFlight=0, closed=true] closing connection
2022-03-22 05:38:32.936 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-60, inFlight=0, closed=true] has already terminated
2022-03-22 05:38:32.936 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
2022-03-22 05:38:32.937 DEBUG 10 --- [uster-worker-75] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:38:32.937 ERROR 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Authentication error during reconnection to /10.233.66.75:9042, scheduling retry in 2000 milliseconds
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.233.66.75:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at com.datastax.driver.core.Connection$8.apply(Connection.java:503)
at com.datastax.driver.core.Connection$8.apply(Connection.java:465)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1479)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 05:38:32.939 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 2000 ms
2022-03-22 05:38:33.216 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:38:33.246 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:38:34.946 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:38:34.964 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-61, inFlight=0, closed=false] Connection established, initializing transport
2022-03-22 05:38:37.857 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:38:40.322 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-61, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2022-03-22 05:38:40.323 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-61, inFlight=0, closed=false] because: [/10.233.66.75:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2022-03-22 05:38:40.323 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-61, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:38:40.323 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-61, inFlight=0, closed=true] closing connection
2022-03-22 05:38:40.323 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:38:40.323 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Not terminating Connection[/10.233.66.75:9042-61, inFlight=0, closed=true]: there are still pending requests
2022-03-22 05:38:40.332 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Connection has been closed), scheduling retry in 4000 milliseconds
2022-03-22 05:38:40.338 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 4000 ms
2022-03-22 05:38:40.457 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.233.66.75:9042, scheduling delivery
2022-03-22 05:38:40.457 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] received event DOWN
2022-03-22 05:38:41.458 DEBUG 10 --- [uster-worker-75] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:38:42.864 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1451815077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27acd25f at consistency null
2022-03-22 05:38:42.864 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1451815077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:42.981 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [331285853-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fa2c2fa at consistency null
2022-03-22 05:38:42.983 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [331285853-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:43.053 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1461367347-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@403134a2 at consistency null
2022-03-22 05:38:43.053 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1461367347-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:43.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [955683603-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5166b0a1 at consistency null
2022-03-22 05:38:43.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [955683603-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:43.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703218032-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1099bef8 at consistency null
2022-03-22 05:38:43.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703218032-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:43.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1108622589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69d51b03 at consistency null
2022-03-22 05:38:43.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1108622589-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:43.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1075680678-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2088b6bf at consistency null
2022-03-22 05:38:43.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1075680678-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:43.946 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1291839677-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@448fe9b7 at consistency null
2022-03-22 05:38:43.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1291839677-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:44.339 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:38:44.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [929415571-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d289478 at consistency null
2022-03-22 05:38:44.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [929415571-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:44.376 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-62, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:38:44.376 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-62, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:38:44.376 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-62, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:38:44.376 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-62, inFlight=0, closed=true] closing connection
2022-03-22 05:38:44.377 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:38:44.377 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 8000 milliseconds
2022-03-22 05:38:44.377 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 8000 ms
2022-03-22 05:38:45.177 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1417011286-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@651003b at consistency null
2022-03-22 05:38:45.177 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1417011286-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:46.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2028597211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@789ec0b0 at consistency null
2022-03-22 05:38:46.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2028597211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:46.808 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:38:51.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1272990331-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27550ff0 at consistency null
2022-03-22 05:38:51.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1272990331-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:52.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1385198541-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ea7c436 at consistency null
2022-03-22 05:38:52.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1385198541-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:52.134 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [698113689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50f95701 at consistency null
2022-03-22 05:38:52.134 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [698113689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:52.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [515701574-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5eec96ee at consistency null
2022-03-22 05:38:52.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [515701574-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:52.378 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:38:52.415 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-63, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:38:52.415 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-63, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:38:52.415 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-63, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:38:52.415 DEBUG 10 --- [uster-worker-75] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:38:52.415 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-63, inFlight=0, closed=true] closing connection
2022-03-22 05:38:52.416 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 16000 milliseconds
2022-03-22 05:38:52.416 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 16000 ms
2022-03-22 05:38:52.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1464256888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51c9384c at consistency null
2022-03-22 05:38:52.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1464256888-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:53.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [110741062-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31afc037 at consistency null
2022-03-22 05:38:53.034 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [110741062-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:53.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1295661210-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bf4c85c at consistency null
2022-03-22 05:38:53.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1295661210-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:53.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [874876080-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1195602a at consistency null
2022-03-22 05:38:53.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [874876080-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:54.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1439276613-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@986210c at consistency null
2022-03-22 05:38:54.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1439276613-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:55.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497731916-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46e7e673 at consistency null
2022-03-22 05:38:55.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497731916-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:38:55.133 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:39:00.167 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [21548171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b5c70db at consistency null
2022-03-22 05:39:00.167 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [21548171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:00.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [428904478-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71f2ddaa at consistency null
2022-03-22 05:39:00.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [428904478-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:00.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [11673311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49c41cd9 at consistency null
2022-03-22 05:39:00.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [11673311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:00.463 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1268340524-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3069afec at consistency null
2022-03-22 05:39:00.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1268340524-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:00.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208860736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@181bcb92 at consistency null
2022-03-22 05:39:00.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208860736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:00.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [301959088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fe35ab6 at consistency null
2022-03-22 05:39:00.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [301959088-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:01.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [19609526-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1628b347 at consistency null
2022-03-22 05:39:01.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [19609526-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:01.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [663688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bc368a1 at consistency null
2022-03-22 05:39:01.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [663688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:01.651 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458298943-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d426910 at consistency null
2022-03-22 05:39:01.651 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458298943-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:02.059 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1996407010-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4891b0e4 at consistency null
2022-03-22 05:39:02.059 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1996407010-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:02.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [774950668-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b70903a at consistency null
2022-03-22 05:39:02.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [774950668-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:02.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2138027042-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b066595 at consistency null
2022-03-22 05:39:02.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2138027042-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:03.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1822205214-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55328b0c at consistency null
2022-03-22 05:39:03.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1822205214-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:03.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1818595775-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@106107de at consistency null
2022-03-22 05:39:03.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1818595775-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:03.834 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:39:08.416 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:39:08.423 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-64, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:39:08.423 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-64, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:39:08.424 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-64, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:39:08.424 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-64, inFlight=0, closed=true] closing connection
2022-03-22 05:39:08.424 DEBUG 10 --- [uster-worker-74] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:39:08.425 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 32000 milliseconds
2022-03-22 05:39:08.425 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 32000 ms
2022-03-22 05:39:08.659 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-61, inFlight=0, closed=true] has already terminated
2022-03-22 05:39:08.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1141205904-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f8fb4a8 at consistency null
2022-03-22 05:39:08.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1141205904-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:08.886 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1179195798-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57fd5201 at consistency null
2022-03-22 05:39:08.887 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1179195798-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:08.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341452308-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e1163fa at consistency null
2022-03-22 05:39:08.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341452308-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:09.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266106940-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c49ac64 at consistency null
2022-03-22 05:39:09.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266106940-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:09.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1617615804-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@419984b6 at consistency null
2022-03-22 05:39:09.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1617615804-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:09.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860303758-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6127038d at consistency null
2022-03-22 05:39:09.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860303758-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:09.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [108909078-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@642cb2e0 at consistency null
2022-03-22 05:39:09.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [108909078-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:10.457 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:39:10.464 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:39:10.548 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1779189629-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55b7188d at consistency null
2022-03-22 05:39:10.548 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1779189629-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:11.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552799419-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58868ab9 at consistency null
2022-03-22 05:39:11.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552799419-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:11.388 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:39:16.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1037138735-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e4d14f at consistency null
2022-03-22 05:39:16.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1037138735-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:16.472 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1431626721-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ed821 at consistency null
2022-03-22 05:39:16.473 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1431626721-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:16.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1575707033-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d40dbd7 at consistency null
2022-03-22 05:39:16.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1575707033-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:16.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1868651665-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ff4784 at consistency null
2022-03-22 05:39:16.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1868651665-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:16.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499417169-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26b6805d at consistency null
2022-03-22 05:39:16.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499417169-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:17.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [728166768-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50dfcb7a at consistency null
2022-03-22 05:39:17.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [728166768-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:17.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [763865644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b2a2cd5 at consistency null
2022-03-22 05:39:17.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [763865644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:18.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223159513-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40129f53 at consistency null
2022-03-22 05:39:18.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223159513-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:20.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1808345174-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52344b64 at consistency null
2022-03-22 05:39:20.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1808345174-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:20.004 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:39:25.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2008594734-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c9ece43 at consistency null
2022-03-22 05:39:25.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2008594734-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:25.236 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1744747327-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16381981 at consistency null
2022-03-22 05:39:25.244 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1744747327-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:25.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270008383-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4204f412 at consistency null
2022-03-22 05:39:25.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270008383-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:25.626 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [850465266-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@321be85d at consistency null
2022-03-22 05:39:25.626 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [850465266-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:25.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15904579-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38fdc48b at consistency null
2022-03-22 05:39:25.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15904579-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:26.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [306694945-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6edf3604 at consistency null
2022-03-22 05:39:26.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [306694945-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:26.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1116616878-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b9efba9 at consistency null
2022-03-22 05:39:26.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1116616878-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:27.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2013958028-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66aab387 at consistency null
2022-03-22 05:39:27.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2013958028-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:27.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1830268793-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3025ea45 at consistency null
2022-03-22 05:39:27.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1830268793-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:28.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1101365049-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27dd8553 at consistency null
2022-03-22 05:39:28.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1101365049-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:28.813 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:39:33.851 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1927111915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5be271b2 at consistency null
2022-03-22 05:39:33.851 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1927111915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:33.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1577575379-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6867c7c2 at consistency null
2022-03-22 05:39:33.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1577575379-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:34.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [247194381-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@778f6f49 at consistency null
2022-03-22 05:39:34.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [247194381-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:34.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1678580052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cb38bf6 at consistency null
2022-03-22 05:39:34.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1678580052-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:34.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2112803586-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74941233 at consistency null
2022-03-22 05:39:34.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2112803586-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:34.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479641111-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@221b82b4 at consistency null
2022-03-22 05:39:34.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479641111-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:34.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298437379-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ee25499 at consistency null
2022-03-22 05:39:34.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298437379-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:34.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1102045762-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e2e8688 at consistency null
2022-03-22 05:39:34.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1102045762-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:35.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1255301046-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a7f08f4 at consistency null
2022-03-22 05:39:35.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1255301046-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:35.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662859005-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@248f2f2d at consistency null
2022-03-22 05:39:35.438 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662859005-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:35.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179858192-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18f5e1b9 at consistency null
2022-03-22 05:39:35.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179858192-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:36.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [876295671-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62952ee9 at consistency null
2022-03-22 05:39:36.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [876295671-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:37.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1825366825-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@400fa2e7 at consistency null
2022-03-22 05:39:37.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1825366825-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:37.621 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:39:40.425 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:39:40.447 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-65, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:39:40.447 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-65, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:39:40.447 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-65, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:39:40.448 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-65, inFlight=0, closed=true] closing connection
2022-03-22 05:39:40.448 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 64000 milliseconds
2022-03-22 05:39:40.449 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 64000 ms
2022-03-22 05:39:40.449 DEBUG 10 --- [uster-worker-76] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:39:40.486 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:39:40.505 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:39:42.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1718393491-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d241b8c at consistency null
2022-03-22 05:39:42.657 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1718393491-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:42.697 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [801480101-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2aebd9b8 at consistency null
2022-03-22 05:39:42.697 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [801480101-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:42.800 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1111261479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6142486a at consistency null
2022-03-22 05:39:42.800 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1111261479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:42.856 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1239611362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c67d63f at consistency null
2022-03-22 05:39:42.856 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1239611362-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:42.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1264075855-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61224e44 at consistency null
2022-03-22 05:39:42.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1264075855-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:43.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215549492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15f07dc1 at consistency null
2022-03-22 05:39:43.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215549492-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:43.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860195748-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6602760d at consistency null
2022-03-22 05:39:43.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860195748-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:43.680 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [820245117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43682dc at consistency null
2022-03-22 05:39:43.680 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [820245117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:44.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [957391378-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7715ac70 at consistency null
2022-03-22 05:39:44.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [957391378-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:44.900 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1865688948-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@611dd926 at consistency null
2022-03-22 05:39:44.900 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1865688948-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:45.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60097091-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14da6916 at consistency null
2022-03-22 05:39:45.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60097091-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:45.735 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:39:50.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743073878-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ee677f6 at consistency null
2022-03-22 05:39:50.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743073878-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:50.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284577791-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@713d55e5 at consistency null
2022-03-22 05:39:50.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284577791-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:51.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1589101842-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36e8000d at consistency null
2022-03-22 05:39:51.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1589101842-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:51.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1973113757-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@431eb442 at consistency null
2022-03-22 05:39:51.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1973113757-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:51.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [174890400-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e8a2210 at consistency null
2022-03-22 05:39:51.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [174890400-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:52.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [675783333-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1503dc59 at consistency null
2022-03-22 05:39:52.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [675783333-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:52.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211661802-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@631d4e9c at consistency null
2022-03-22 05:39:52.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211661802-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:53.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1729153227-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55175a19 at consistency null
2022-03-22 05:39:53.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1729153227-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:54.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [167761940-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a527809 at consistency null
2022-03-22 05:39:54.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [167761940-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:54.035 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:39:59.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [473432324-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1388c0e at consistency null
2022-03-22 05:39:59.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [473432324-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:59.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [392489028-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8d85b1b at consistency null
2022-03-22 05:39:59.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [392489028-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:59.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1993551091-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67700f8b at consistency null
2022-03-22 05:39:59.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1993551091-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:59.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18576709-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e7ba15c at consistency null
2022-03-22 05:39:59.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18576709-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:59.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1568105539-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b661a61 at consistency null
2022-03-22 05:39:59.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1568105539-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:39:59.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835106657-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71ac8e95 at consistency null
2022-03-22 05:39:59.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835106657-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:00.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [45780106-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1647db7c at consistency null
2022-03-22 05:40:00.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [45780106-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:01.143 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1490116805-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3515f9a9 at consistency null
2022-03-22 05:40:01.143 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1490116805-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:02.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1046631136-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e2b4878 at consistency null
2022-03-22 05:40:02.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1046631136-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:02.803 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:40:07.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1600763588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2be6b10e at consistency null
2022-03-22 05:40:07.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1600763588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:07.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [157400959-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d21335 at consistency null
2022-03-22 05:40:07.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [157400959-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:07.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524969950-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19106a57 at consistency null
2022-03-22 05:40:07.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524969950-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:07.998 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1983679792-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d1c9985 at consistency null
2022-03-22 05:40:07.998 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1983679792-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:08.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693571749-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@636c29e9 at consistency null
2022-03-22 05:40:08.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693571749-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:08.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626282755-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7034c50a at consistency null
2022-03-22 05:40:08.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626282755-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:08.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [326614273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a6c4b89 at consistency null
2022-03-22 05:40:08.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [326614273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:08.473 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [19986258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73a537f0 at consistency null
2022-03-22 05:40:08.473 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [19986258-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:08.694 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1803432227-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@274eecb3 at consistency null
2022-03-22 05:40:08.694 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1803432227-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:08.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1934694476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ff6dbd0 at consistency null
2022-03-22 05:40:08.959 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1934694476-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:09.176 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1574972562-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75366369 at consistency null
2022-03-22 05:40:09.176 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1574972562-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:09.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1533860379-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27383ac8 at consistency null
2022-03-22 05:40:09.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1533860379-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:10.044 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098981244-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@774361e8 at consistency null
2022-03-22 05:40:10.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098981244-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:10.467 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [980020344-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e721c8b at consistency null
2022-03-22 05:40:10.467 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [980020344-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:10.506 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:40:10.522 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:40:11.291 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1714537243-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28991707 at consistency null
2022-03-22 05:40:11.291 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1714537243-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:11.294 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:435)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:40:16.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1529901508-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e9196a6 at consistency null
2022-03-22 05:40:16.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1529901508-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:16.366 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310636321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22b1f2bd at consistency null
2022-03-22 05:40:16.367 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310636321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:16.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736231717-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cd92424 at consistency null
2022-03-22 05:40:16.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736231717-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:16.490 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264103370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c983197 at consistency null
2022-03-22 05:40:16.490 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264103370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:16.595 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1465584552-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e7ea404 at consistency null
2022-03-22 05:40:16.596 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1465584552-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:16.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1521228933-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@461a7332 at consistency null
2022-03-22 05:40:16.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1521228933-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:16.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535693078-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30a63b01 at consistency null
2022-03-22 05:40:16.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535693078-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:17.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1006369816-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f95948c at consistency null
2022-03-22 05:40:17.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1006369816-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:18.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879928964-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a03ffaf at consistency null
2022-03-22 05:40:18.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879928964-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:19.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [234937531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ee3bc8e at consistency null
2022-03-22 05:40:19.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [234937531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:19.935 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:40:25.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [320715788-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f9cfc60 at consistency null
2022-03-22 05:40:25.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [320715788-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:25.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2002950475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c5e34eb at consistency null
2022-03-22 05:40:25.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2002950475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:25.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1451579301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49f55e48 at consistency null
2022-03-22 05:40:25.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1451579301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:25.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1583918282-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a818c34 at consistency null
2022-03-22 05:40:25.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1583918282-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:25.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [543921879-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@327618f8 at consistency null
2022-03-22 05:40:25.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [543921879-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:25.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134623568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fe28b31 at consistency null
2022-03-22 05:40:25.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134623568-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:26.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063564542-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c86999b at consistency null
2022-03-22 05:40:26.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063564542-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:26.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619794724-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ad05213 at consistency null
2022-03-22 05:40:26.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619794724-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:27.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1481680451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8e32a69 at consistency null
2022-03-22 05:40:27.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1481680451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:28.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [477015138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43139347 at consistency null
2022-03-22 05:40:28.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [477015138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:28.128 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:40:33.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [104278101-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b5190ff at consistency null
2022-03-22 05:40:33.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [104278101-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:33.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [685846982-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4319134c at consistency null
2022-03-22 05:40:33.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [685846982-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:33.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1969154087-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52b8c924 at consistency null
2022-03-22 05:40:33.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1969154087-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:33.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [473581394-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@deee19f at consistency null
2022-03-22 05:40:33.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [473581394-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:33.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [594583808-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4aff4eb5 at consistency null
2022-03-22 05:40:33.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [594583808-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:33.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [37673897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41f7b7d5 at consistency null
2022-03-22 05:40:33.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [37673897-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:34.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2294809-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5592b726 at consistency null
2022-03-22 05:40:34.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2294809-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:34.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315816344-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f8a4612 at consistency null
2022-03-22 05:40:34.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315816344-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:34.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1748173812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fb03fb0 at consistency null
2022-03-22 05:40:34.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1748173812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:35.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [480538312-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63856a8c at consistency null
2022-03-22 05:40:35.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [480538312-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:36.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [71283337-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@236eca5d at consistency null
2022-03-22 05:40:36.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [71283337-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:36.855 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619626645-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e29b2f4 at consistency null
2022-03-22 05:40:36.856 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619626645-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:36.864 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:40:40.537 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:40:40.540 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:40:41.885 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420974943-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40fa4b3e at consistency null
2022-03-22 05:40:41.885 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420974943-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:41.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809825071-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@86388a7 at consistency null
2022-03-22 05:40:41.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809825071-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:41.946 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [311288103-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d8a87d4 at consistency null
2022-03-22 05:40:41.946 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [311288103-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:42.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1085632907-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e272083 at consistency null
2022-03-22 05:40:42.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1085632907-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:42.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [110867041-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c9631c2 at consistency null
2022-03-22 05:40:42.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [110867041-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:42.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [272318808-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4824ac47 at consistency null
2022-03-22 05:40:42.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [272318808-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:42.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1833045665-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@419926b0 at consistency null
2022-03-22 05:40:42.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1833045665-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:43.136 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [110105515-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36cd2538 at consistency null
2022-03-22 05:40:43.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [110105515-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:43.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564717474-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b380a02 at consistency null
2022-03-22 05:40:43.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564717474-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:44.451 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:40:44.456 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-66, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:40:44.457 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-66, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:40:44.457 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-66, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:40:44.459 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-66, inFlight=0, closed=true] closing connection
2022-03-22 05:40:44.459 DEBUG 10 --- [uster-worker-77] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:40:44.459 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 128000 milliseconds
2022-03-22 05:40:44.460 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 128000 ms
2022-03-22 05:40:45.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389548755-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7affa750 at consistency null
2022-03-22 05:40:45.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389548755-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:45.547 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:40:50.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1785870739-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52d9d5da at consistency null
2022-03-22 05:40:50.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1785870739-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:50.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1819898429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5295dcac at consistency null
2022-03-22 05:40:50.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1819898429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:50.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754389169-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55559006 at consistency null
2022-03-22 05:40:50.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754389169-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:51.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2068769134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@529768db at consistency null
2022-03-22 05:40:51.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2068769134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:51.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [780597-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32f0d599 at consistency null
2022-03-22 05:40:51.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [780597-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:51.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [658807699-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b5cb661 at consistency null
2022-03-22 05:40:51.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [658807699-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:51.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866280786-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22e4ce61 at consistency null
2022-03-22 05:40:51.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866280786-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:52.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1059848145-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c3494db at consistency null
2022-03-22 05:40:52.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1059848145-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:53.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1614211904-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@385cc4d9 at consistency null
2022-03-22 05:40:53.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1614211904-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:53.006 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:40:58.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1663921366-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@169d820d at consistency null
2022-03-22 05:40:58.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1663921366-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:58.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1132797204-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@128ec179 at consistency null
2022-03-22 05:40:58.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1132797204-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:58.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598783938-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a341dbf at consistency null
2022-03-22 05:40:58.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598783938-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:58.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1955275807-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64106c2c at consistency null
2022-03-22 05:40:58.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1955275807-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:58.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2093526404-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ee70749 at consistency null
2022-03-22 05:40:58.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2093526404-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:58.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [399401652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a0688e5 at consistency null
2022-03-22 05:40:58.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [399401652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:58.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [837212456-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6045bbff at consistency null
2022-03-22 05:40:58.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [837212456-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:59.132 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484489431-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d3dad7 at consistency null
2022-03-22 05:40:59.132 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484489431-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:40:59.944 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [637375042-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2128482b at consistency null
2022-03-22 05:40:59.944 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [637375042-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:01.552 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1103273867-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f27ad4b at consistency null
2022-03-22 05:41:01.552 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1103273867-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:01.554 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:41:06.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181360386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22707913 at consistency null
2022-03-22 05:41:06.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181360386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:06.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599414040-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bec0ae8 at consistency null
2022-03-22 05:41:06.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599414040-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:06.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [376695090-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58e674ae at consistency null
2022-03-22 05:41:06.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [376695090-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:06.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1125647024-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@af9c70c at consistency null
2022-03-22 05:41:06.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1125647024-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:06.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1424837088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b609fa1 at consistency null
2022-03-22 05:41:06.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1424837088-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:07.045 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [590819770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70806b6f at consistency null
2022-03-22 05:41:07.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [590819770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:07.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [704935575-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ba822d5 at consistency null
2022-03-22 05:41:07.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [704935575-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:07.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [688585957-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fad470b at consistency null
2022-03-22 05:41:07.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [688585957-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:08.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [395711085-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@701f56c8 at consistency null
2022-03-22 05:41:08.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [395711085-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:08.900 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [230050756-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b780b7 at consistency null
2022-03-22 05:41:08.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [230050756-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:09.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68101163-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5953a49e at consistency null
2022-03-22 05:41:09.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68101163-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:09.713 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:41:10.555 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:41:10.563 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:41:14.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2057047451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c7335a0 at consistency null
2022-03-22 05:41:14.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2057047451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:14.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564372447-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66d087f0 at consistency null
2022-03-22 05:41:14.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564372447-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:14.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1659495895-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@595d01a5 at consistency null
2022-03-22 05:41:14.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1659495895-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:14.869 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523942877-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d6e2944 at consistency null
2022-03-22 05:41:14.869 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523942877-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:14.978 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [97553780-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6845c17c at consistency null
2022-03-22 05:41:14.978 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [97553780-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:15.084 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724457604-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a483483 at consistency null
2022-03-22 05:41:15.084 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724457604-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:15.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [113559403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ba94c46 at consistency null
2022-03-22 05:41:15.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [113559403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:15.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282043965-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57e1767e at consistency null
2022-03-22 05:41:15.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282043965-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:15.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1305538537-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@116903df at consistency null
2022-03-22 05:41:15.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1305538537-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:15.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [342062255-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5acec15d at consistency null
2022-03-22 05:41:15.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [342062255-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:15.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [884741847-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@777abcb8 at consistency null
2022-03-22 05:41:15.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [884741847-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:16.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [726164453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e543e73 at consistency null
2022-03-22 05:41:16.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [726164453-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:17.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630367426-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bcd6cd at consistency null
2022-03-22 05:41:17.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630367426-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:18.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860483140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@aa94629 at consistency null
2022-03-22 05:41:18.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860483140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:18.712 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:41:24.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [976880097-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2894a4b8 at consistency null
2022-03-22 05:41:24.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [976880097-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:24.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1376066056-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27bcb07f at consistency null
2022-03-22 05:41:24.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1376066056-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:24.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1490302452-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@435200fb at consistency null
2022-03-22 05:41:24.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1490302452-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:24.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [322539570-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38a8eb6 at consistency null
2022-03-22 05:41:24.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [322539570-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:24.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045161149-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e364b1b at consistency null
2022-03-22 05:41:24.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045161149-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:25.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [55131859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ccb2dea at consistency null
2022-03-22 05:41:25.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [55131859-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:25.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2051784923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fcda950 at consistency null
2022-03-22 05:41:25.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2051784923-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:25.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [623667104-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48e56b21 at consistency null
2022-03-22 05:41:25.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [623667104-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:25.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1769191166-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74b78fb5 at consistency null
2022-03-22 05:41:25.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1769191166-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:26.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [587641509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@610648b4 at consistency null
2022-03-22 05:41:26.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [587641509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:27.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [133161734-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13f3faed at consistency null
2022-03-22 05:41:27.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [133161734-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:27.131 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:41:32.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1102611772-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70ac37d1 at consistency null
2022-03-22 05:41:32.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1102611772-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:32.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889724851-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@382ed456 at consistency null
2022-03-22 05:41:32.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889724851-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:32.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596443181-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bbe780f at consistency null
2022-03-22 05:41:32.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596443181-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:32.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535258761-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6796d307 at consistency null
2022-03-22 05:41:32.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535258761-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:32.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578553248-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7695629b at consistency null
2022-03-22 05:41:32.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578553248-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:32.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [722908758-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3857947b at consistency null
2022-03-22 05:41:32.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [722908758-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:32.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1615939441-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2031fd7e at consistency null
2022-03-22 05:41:32.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1615939441-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:32.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [706557322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@716e0c94 at consistency null
2022-03-22 05:41:32.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [706557322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:33.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1117688998-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cb22998 at consistency null
2022-03-22 05:41:33.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1117688998-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:33.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1545875109-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28428439 at consistency null
2022-03-22 05:41:33.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1545875109-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:33.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916754591-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b74642 at consistency null
2022-03-22 05:41:33.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916754591-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:33.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1976976036-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63cf08c at consistency null
2022-03-22 05:41:33.744 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1976976036-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:34.148 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138475804-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bffe73a at consistency null
2022-03-22 05:41:34.148 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138475804-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:34.573 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1048022212-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@720cf30 at consistency null
2022-03-22 05:41:34.573 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1048022212-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:34.991 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [742769661-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a980cc0 at consistency null
2022-03-22 05:41:34.991 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [742769661-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:35.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1807837107-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bc1338f at consistency null
2022-03-22 05:41:35.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1807837107-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:35.404 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:41:40.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981778676-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fc3c9fa at consistency null
2022-03-22 05:41:40.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981778676-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:40.486 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1955346783-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e1bcc7d at consistency null
2022-03-22 05:41:40.486 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1955346783-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:40.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1993627160-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@649f3d72 at consistency null
2022-03-22 05:41:40.556 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1993627160-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:40.565 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:41:40.570 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:41:40.664 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [373836686-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36902725 at consistency null
2022-03-22 05:41:40.665 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [373836686-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:40.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [478821273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@379db414 at consistency null
2022-03-22 05:41:40.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [478821273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:41.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156876694-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7725febe at consistency null
2022-03-22 05:41:41.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156876694-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:41.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1567185971-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dd292f6 at consistency null
2022-03-22 05:41:41.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1567185971-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:42.135 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [162097195-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b5b04c at consistency null
2022-03-22 05:41:42.135 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [162097195-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:42.565 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1152568436-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f3d6e0a at consistency null
2022-03-22 05:41:42.565 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1152568436-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:43.387 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5467771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5436e469 at consistency null
2022-03-22 05:41:43.387 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5467771-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:43.390 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:41:48.396 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390865542-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dba993b at consistency null
2022-03-22 05:41:48.396 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390865542-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:48.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1441196148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37277d10 at consistency null
2022-03-22 05:41:48.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1441196148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:48.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [132468688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3de18c0 at consistency null
2022-03-22 05:41:48.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [132468688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:48.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475978209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60a8a17e at consistency null
2022-03-22 05:41:48.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475978209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:48.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794146558-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29129f11 at consistency null
2022-03-22 05:41:48.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794146558-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:48.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [146235548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2551b8de at consistency null
2022-03-22 05:41:48.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [146235548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:49.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [620832149-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56babcac at consistency null
2022-03-22 05:41:49.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [620832149-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:49.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341812997-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@733c5454 at consistency null
2022-03-22 05:41:49.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341812997-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:49.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568586100-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4eebb735 at consistency null
2022-03-22 05:41:49.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568586100-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:50.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [308533332-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66d4717e at consistency null
2022-03-22 05:41:50.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [308533332-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:50.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1055029760-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6588fed4 at consistency null
2022-03-22 05:41:50.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1055029760-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:50.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579618432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@740798c5 at consistency null
2022-03-22 05:41:50.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579618432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:51.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472548842-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67c6db6f at consistency null
2022-03-22 05:41:51.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472548842-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:51.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [277303107-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@225c1968 at consistency null
2022-03-22 05:41:51.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [277303107-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:52.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1024988457-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3da9 at consistency null
2022-03-22 05:41:52.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1024988457-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:52.010 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:41:57.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103503948-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48894cdc at consistency null
2022-03-22 05:41:57.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103503948-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:57.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [900539359-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1945fb01 at consistency null
2022-03-22 05:41:57.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [900539359-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:57.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536514430-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b102618 at consistency null
2022-03-22 05:41:57.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536514430-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:57.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1656123637-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5afcb5a6 at consistency null
2022-03-22 05:41:57.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1656123637-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:57.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407827043-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5811fd89 at consistency null
2022-03-22 05:41:57.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407827043-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:57.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151437166-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e3a9879 at consistency null
2022-03-22 05:41:57.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151437166-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:57.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258699259-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34c04340 at consistency null
2022-03-22 05:41:57.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258699259-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:58.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1586206523-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b2e91fb at consistency null
2022-03-22 05:41:58.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1586206523-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:58.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [834251619-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c04c8a2 at consistency null
2022-03-22 05:41:58.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [834251619-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:41:59.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040245955-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66c76a0b at consistency null
2022-03-22 05:41:59.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040245955-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:01.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [760939196-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@502bffd0 at consistency null
2022-03-22 05:42:01.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [760939196-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:01.105 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:42:06.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795568739-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f751d67 at consistency null
2022-03-22 05:42:06.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795568739-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:06.178 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1408577118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@554e91f3 at consistency null
2022-03-22 05:42:06.178 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1408577118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:06.284 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2008837973-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8842e63 at consistency null
2022-03-22 05:42:06.284 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2008837973-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:06.497 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [42454524-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75e95dc at consistency null
2022-03-22 05:42:06.497 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [42454524-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:06.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1984661437-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c73e75 at consistency null
2022-03-22 05:42:06.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1984661437-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:07.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [213303540-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21a7e697 at consistency null
2022-03-22 05:42:07.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [213303540-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:07.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417648880-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8f923f6 at consistency null
2022-03-22 05:42:07.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417648880-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:07.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607131197-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b7ab740 at consistency null
2022-03-22 05:42:07.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607131197-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:08.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250350441-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42f70acc at consistency null
2022-03-22 05:42:08.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250350441-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:09.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879217440-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c7acf5b at consistency null
2022-03-22 05:42:09.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879217440-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:09.538 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:42:10.572 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:42:10.579 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:42:14.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171634188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dddb1b1 at consistency null
2022-03-22 05:42:14.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171634188-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:14.572 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [773816456-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3333c5fd at consistency null
2022-03-22 05:42:14.572 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [773816456-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:14.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1078358369-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4968083d at consistency null
2022-03-22 05:42:14.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1078358369-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:14.659 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1257413762-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43e2065c at consistency null
2022-03-22 05:42:14.659 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1257413762-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:14.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [400904542-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@334267a8 at consistency null
2022-03-22 05:42:14.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [400904542-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:14.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1160546538-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ef34010 at consistency null
2022-03-22 05:42:14.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1160546538-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:15.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907158615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17538a13 at consistency null
2022-03-22 05:42:15.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907158615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:15.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1759164571-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44d0ae43 at consistency null
2022-03-22 05:42:15.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1759164571-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:15.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1435003775-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@640fee56 at consistency null
2022-03-22 05:42:15.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1435003775-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:16.049 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950532348-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6078d107 at consistency null
2022-03-22 05:42:16.050 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950532348-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:16.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406492863-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b70dc09 at consistency null
2022-03-22 05:42:16.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406492863-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:17.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1861616702-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a93d592 at consistency null
2022-03-22 05:42:17.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1861616702-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:18.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025019660-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31f5e3c8 at consistency null
2022-03-22 05:42:18.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025019660-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:18.123 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:42:23.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [444867012-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1723ec18 at consistency null
2022-03-22 05:42:23.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [444867012-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:23.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1647078067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7adefa61 at consistency null
2022-03-22 05:42:23.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1647078067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:23.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1017464124-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ed59fed at consistency null
2022-03-22 05:42:23.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1017464124-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:23.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1580990621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5020ce5c at consistency null
2022-03-22 05:42:23.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1580990621-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:23.663 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703776415-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58ebbb6c at consistency null
2022-03-22 05:42:23.663 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703776415-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:23.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693779823-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55eb4fe2 at consistency null
2022-03-22 05:42:23.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693779823-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:24.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919890024-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37a39845 at consistency null
2022-03-22 05:42:24.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919890024-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:24.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282014288-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6127e2c3 at consistency null
2022-03-22 05:42:24.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282014288-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:24.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172931626-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64121550 at consistency null
2022-03-22 05:42:24.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172931626-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:24.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [158023047-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b89c26f at consistency null
2022-03-22 05:42:24.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [158023047-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:24.657 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [35435184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40c30eeb at consistency null
2022-03-22 05:42:24.657 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [35435184-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:24.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18109919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e2cddca at consistency null
2022-03-22 05:42:24.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18109919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:25.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [148195587-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ae0f3ca at consistency null
2022-03-22 05:42:25.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [148195587-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:25.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1476577947-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fdfde43 at consistency null
2022-03-22 05:42:25.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1476577947-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:26.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1072462361-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17b32505 at consistency null
2022-03-22 05:42:26.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1072462361-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:26.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [485845654-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f0968b4 at consistency null
2022-03-22 05:42:26.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [485845654-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:26.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1053472090-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21ab83c at consistency null
2022-03-22 05:42:26.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1053472090-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:26.916 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:42:31.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1892873738-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d13c392 at consistency null
2022-03-22 05:42:31.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1892873738-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:32.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [749893307-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@197b504c at consistency null
2022-03-22 05:42:32.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [749893307-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:32.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [455346273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e77c007 at consistency null
2022-03-22 05:42:32.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [455346273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:32.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1882132026-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fefd3b0 at consistency null
2022-03-22 05:42:32.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1882132026-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:32.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397067411-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cc3868c at consistency null
2022-03-22 05:42:32.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397067411-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:33.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [672300915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f794268 at consistency null
2022-03-22 05:42:33.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [672300915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:33.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [323886352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ffec878 at consistency null
2022-03-22 05:42:33.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [323886352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:34.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [454598557-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ebdf264 at consistency null
2022-03-22 05:42:34.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [454598557-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:35.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1515671089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e3afed9 at consistency null
2022-03-22 05:42:35.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1515671089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:35.433 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:42:40.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1627741167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bb58574 at consistency null
2022-03-22 05:42:40.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1627741167-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:40.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46165058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52213f72 at consistency null
2022-03-22 05:42:40.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46165058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:40.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576796144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@530ec980 at consistency null
2022-03-22 05:42:40.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576796144-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:40.601 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:42:40.607 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:42:40.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1035797688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a8b390e at consistency null
2022-03-22 05:42:40.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1035797688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:40.696 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982655171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dc2810e at consistency null
2022-03-22 05:42:40.696 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982655171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:40.760 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2005871160-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1643ed6b at consistency null
2022-03-22 05:42:40.760 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2005871160-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:40.890 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1971932998-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@332474ea at consistency null
2022-03-22 05:42:40.890 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1971932998-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:41.099 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541330827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@702468c9 at consistency null
2022-03-22 05:42:41.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541330827-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:41.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [833589682-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29374d32 at consistency null
2022-03-22 05:42:41.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [833589682-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:42.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1073718101-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2096d5ca at consistency null
2022-03-22 05:42:42.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1073718101-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:43.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [11745427-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30856616 at consistency null
2022-03-22 05:42:43.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [11745427-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:43.157 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:42:48.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1787227-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@633422ef at consistency null
2022-03-22 05:42:48.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1787227-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:48.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1299478251-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70f9bd66 at consistency null
2022-03-22 05:42:48.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1299478251-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:48.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [265519387-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66d70e36 at consistency null
2022-03-22 05:42:48.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [265519387-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:48.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [373114834-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d79612c at consistency null
2022-03-22 05:42:48.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [373114834-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:48.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1280764371-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f181736 at consistency null
2022-03-22 05:42:48.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1280764371-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:48.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [586878364-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b78f3e at consistency null
2022-03-22 05:42:48.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [586878364-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:49.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825461633-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ccf87e0 at consistency null
2022-03-22 05:42:49.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825461633-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:50.238 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1305237922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1557e684 at consistency null
2022-03-22 05:42:50.238 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1305237922-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:51.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [909060537-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ba7556f at consistency null
2022-03-22 05:42:51.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [909060537-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:42:51.124 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:35:00Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:42:52.482 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:42:52.489 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-67, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:42:52.489 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-67, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:42:52.489 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-67, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:42:52.507 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-67, inFlight=0, closed=true] closing connection
2022-03-22 05:42:52.508 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 256000 milliseconds
2022-03-22 05:42:52.508 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 256000 ms
2022-03-22 05:42:52.518 DEBUG 10 --- [uster-worker-78] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:43:10.615 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:43:10.618 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:43:40.644 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:43:40.649 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:43:44.228 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:29.502Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 26 common frames omitted
2022-03-22 05:44:06.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [438306779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b77277c at consistency null
2022-03-22 05:44:06.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [438306779-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:06.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1471394806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e31981b at consistency null
2022-03-22 05:44:06.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1471394806-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:06.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [507045274-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f8289f at consistency null
2022-03-22 05:44:06.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [507045274-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:06.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [748062918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15e50942 at consistency null
2022-03-22 05:44:06.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [748062918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:07.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907172944-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@271b8045 at consistency null
2022-03-22 05:44:07.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907172944-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:07.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088007015-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c825fd at consistency null
2022-03-22 05:44:07.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088007015-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:07.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632904518-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d23e746 at consistency null
2022-03-22 05:44:07.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632904518-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:08.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1030825871-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@641590f at consistency null
2022-03-22 05:44:08.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1030825871-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:09.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421862573-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@426273f7 at consistency null
2022-03-22 05:44:09.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421862573-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:10.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176866211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8d113b1 at consistency null
2022-03-22 05:44:10.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176866211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:10.323 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:44:10.672 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:44:10.677 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:44:15.367 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [690593098-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6aaadc3a at consistency null
2022-03-22 05:44:15.367 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [690593098-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:15.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [491821759-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@662f7941 at consistency null
2022-03-22 05:44:15.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [491821759-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:15.491 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [748764620-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@370343b8 at consistency null
2022-03-22 05:44:15.491 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [748764620-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:15.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1174664558-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37cf8272 at consistency null
2022-03-22 05:44:15.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1174664558-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:15.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589689641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bb8c3bd at consistency null
2022-03-22 05:44:15.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589689641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:15.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985202832-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@164056fa at consistency null
2022-03-22 05:44:15.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985202832-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:15.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [525534550-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bf3c909 at consistency null
2022-03-22 05:44:15.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [525534550-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:16.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1185033047-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70641017 at consistency null
2022-03-22 05:44:16.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1185033047-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:16.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1251091473-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d0c808d at consistency null
2022-03-22 05:44:16.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1251091473-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:16.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [945619027-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29eff650 at consistency null
2022-03-22 05:44:16.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [945619027-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:16.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067703227-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c71c9e5 at consistency null
2022-03-22 05:44:16.834 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067703227-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:17.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1745663883-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c4584d4 at consistency null
2022-03-22 05:44:17.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1745663883-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:17.463 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [870687799-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5147f39 at consistency null
2022-03-22 05:44:17.463 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [870687799-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:17.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1676305792-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@531f44bd at consistency null
2022-03-22 05:44:17.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1676305792-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:18.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1119163461-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51a381da at consistency null
2022-03-22 05:44:18.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1119163461-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:19.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1325941623-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2130cc1e at consistency null
2022-03-22 05:44:19.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1325941623-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:19.224 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:44:24.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1372754621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fb82b6 at consistency null
2022-03-22 05:44:24.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1372754621-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:24.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1629764450-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6308ed0e at consistency null
2022-03-22 05:44:24.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1629764450-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:24.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [198777919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@447f9d4d at consistency null
2022-03-22 05:44:24.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [198777919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:24.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [113763784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fca5946 at consistency null
2022-03-22 05:44:24.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [113763784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:24.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1128646871-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71ee4423 at consistency null
2022-03-22 05:44:24.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1128646871-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:24.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [66721014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@552ab78d at consistency null
2022-03-22 05:44:24.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [66721014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:25.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082247250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d87cd90 at consistency null
2022-03-22 05:44:25.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082247250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:25.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2043776047-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@204ab90f at consistency null
2022-03-22 05:44:25.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2043776047-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:25.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [348119218-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23862e79 at consistency null
2022-03-22 05:44:25.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [348119218-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:26.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1194396535-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2390635b at consistency null
2022-03-22 05:44:26.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1194396535-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:27.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [163354779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70a8d1f4 at consistency null
2022-03-22 05:44:27.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [163354779-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:28.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088361335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55f9c2f at consistency null
2022-03-22 05:44:28.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088361335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:28.004 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:44:33.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396892218-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17b88f1b at consistency null
2022-03-22 05:44:33.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396892218-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:33.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1477296907-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3712bc0c at consistency null
2022-03-22 05:44:33.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1477296907-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:33.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899211473-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@103ec6b at consistency null
2022-03-22 05:44:33.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899211473-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:33.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129503600-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7844b680 at consistency null
2022-03-22 05:44:33.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129503600-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:33.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1351000181-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b573ec4 at consistency null
2022-03-22 05:44:33.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1351000181-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:33.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1713714421-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@907157e at consistency null
2022-03-22 05:44:33.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1713714421-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:33.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [340381320-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1779d081 at consistency null
2022-03-22 05:44:33.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [340381320-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:33.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396539104-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a548458 at consistency null
2022-03-22 05:44:33.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396539104-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:33.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [912124410-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@586732c8 at consistency null
2022-03-22 05:44:33.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [912124410-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:33.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2039542104-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@697a605f at consistency null
2022-03-22 05:44:33.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2039542104-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:34.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [595021164-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30de4829 at consistency null
2022-03-22 05:44:34.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [595021164-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:34.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461761948-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e467c2e at consistency null
2022-03-22 05:44:34.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461761948-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:35.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112462639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f56e45c at consistency null
2022-03-22 05:44:35.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112462639-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:35.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1017837071-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5940371 at consistency null
2022-03-22 05:44:35.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1017837071-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:36.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632279259-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47c98cea at consistency null
2022-03-22 05:44:36.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632279259-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:36.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [301080831-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a832f31 at consistency null
2022-03-22 05:44:36.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [301080831-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:36.539 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:44:40.701 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:44:40.712 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:44:41.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622946371-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@227f4a39 at consistency null
2022-03-22 05:44:41.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622946371-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:41.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1322317926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d54a8c at consistency null
2022-03-22 05:44:41.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1322317926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:41.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [354599023-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c703840 at consistency null
2022-03-22 05:44:41.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [354599023-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:41.859 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [358084588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f396458 at consistency null
2022-03-22 05:44:41.859 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [358084588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:42.072 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [991620725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67cfdf04 at consistency null
2022-03-22 05:44:42.072 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [991620725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:42.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1916805816-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ae3785b at consistency null
2022-03-22 05:44:42.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1916805816-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:42.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1902290215-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47dceb7f at consistency null
2022-03-22 05:44:42.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1902290215-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:43.138 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1533025384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33b35c78 at consistency null
2022-03-22 05:44:43.141 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1533025384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:43.558 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [929376311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a9d92f5 at consistency null
2022-03-22 05:44:43.558 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [929376311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:43.967 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1153665409-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b6d165d at consistency null
2022-03-22 05:44:43.967 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1153665409-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:44.791 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [543971235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b4a9d5c at consistency null
2022-03-22 05:44:44.791 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [543971235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:44.792 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:44:49.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2067045530-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72b35ad1 at consistency null
2022-03-22 05:44:49.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2067045530-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:50.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703815003-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@347f7448 at consistency null
2022-03-22 05:44:50.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703815003-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:50.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1029256682-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@289c9aa3 at consistency null
2022-03-22 05:44:50.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1029256682-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:50.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2135604397-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@409138b3 at consistency null
2022-03-22 05:44:50.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2135604397-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:50.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541405074-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f83c9dd at consistency null
2022-03-22 05:44:50.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541405074-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:50.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879063102-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3216133a at consistency null
2022-03-22 05:44:50.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879063102-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:51.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1088939325-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e13de59 at consistency null
2022-03-22 05:44:51.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1088939325-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:51.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1784845912-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26d2cf95 at consistency null
2022-03-22 05:44:51.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1784845912-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:52.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674714425-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24df87b6 at consistency null
2022-03-22 05:44:52.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674714425-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:53.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2083087869-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44de6462 at consistency null
2022-03-22 05:44:53.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2083087869-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:53.539 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:44:58.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60340552-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ba2b14 at consistency null
2022-03-22 05:44:58.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60340552-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:58.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100703029-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32a4e2b1 at consistency null
2022-03-22 05:44:58.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100703029-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:58.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [243195519-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a1e2164 at consistency null
2022-03-22 05:44:58.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [243195519-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:58.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969884864-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d54b531 at consistency null
2022-03-22 05:44:58.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969884864-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:59.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2145118280-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e65d704 at consistency null
2022-03-22 05:44:59.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2145118280-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:59.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [869805825-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b411d13 at consistency null
2022-03-22 05:44:59.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [869805825-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:59.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261273635-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f0a1e at consistency null
2022-03-22 05:44:59.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261273635-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:44:59.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [143075010-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@784aca1b at consistency null
2022-03-22 05:44:59.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [143075010-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:00.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683546162-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5837ffe1 at consistency null
2022-03-22 05:45:00.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683546162-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:01.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1525449731-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b8dd871 at consistency null
2022-03-22 05:45:01.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1525449731-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:01.531 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:45:06.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2113350119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33ea34e4 at consistency null
2022-03-22 05:45:06.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2113350119-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:06.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282908589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d5e3a90 at consistency null
2022-03-22 05:45:06.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282908589-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:06.733 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:45:06.742 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-44, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:45:06.742 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-44, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:45:06.742 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-44, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:45:06.743 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-44, inFlight=0, closed=true] closing connection
2022-03-22 05:45:06.744 DEBUG 10 --- [uster-worker-79] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:45:06.745 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 05:45:06.745 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 600000 ms
2022-03-22 05:45:06.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [337650071-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a71162d at consistency null
2022-03-22 05:45:06.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [337650071-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:06.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718435212-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f1a0742 at consistency null
2022-03-22 05:45:06.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718435212-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:07.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1856954144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d355551 at consistency null
2022-03-22 05:45:07.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1856954144-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:07.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [522547601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51d362dd at consistency null
2022-03-22 05:45:07.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [522547601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:07.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [169658235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23e1d6b4 at consistency null
2022-03-22 05:45:07.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [169658235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:07.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [871078823-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c4ef59d at consistency null
2022-03-22 05:45:07.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [871078823-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:08.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [723780901-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b46b298 at consistency null
2022-03-22 05:45:08.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [723780901-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:08.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532238377-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7149eaa7 at consistency null
2022-03-22 05:45:08.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532238377-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:09.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936878522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cb0c64c at consistency null
2022-03-22 05:45:09.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936878522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:10.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [481026256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@709225a at consistency null
2022-03-22 05:45:10.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [481026256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:10.139 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:45:10.714 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:45:10.719 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:45:15.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397263693-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9559251 at consistency null
2022-03-22 05:45:15.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397263693-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:15.199 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1111548297-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37775d0e at consistency null
2022-03-22 05:45:15.199 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1111548297-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:15.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [571436480-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6daf7f0a at consistency null
2022-03-22 05:45:15.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [571436480-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:15.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1642181681-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@237a9b17 at consistency null
2022-03-22 05:45:15.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1642181681-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:15.752 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [405779691-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6185233 at consistency null
2022-03-22 05:45:15.752 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [405779691-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:16.165 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571177166-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b0ae0d3 at consistency null
2022-03-22 05:45:16.165 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571177166-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:16.582 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [69092823-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d4b8997 at consistency null
2022-03-22 05:45:16.582 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [69092823-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:17.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1756506457-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a9e1b0a at consistency null
2022-03-22 05:45:17.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1756506457-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:18.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [232734792-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@720334d1 at consistency null
2022-03-22 05:45:18.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [232734792-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:18.330 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:45:23.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644221199-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33c51c37 at consistency null
2022-03-22 05:45:23.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644221199-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:23.539 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [340132453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2999170e at consistency null
2022-03-22 05:45:23.539 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [340132453-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:23.630 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1833371989-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dd3a9d8 at consistency null
2022-03-22 05:45:23.630 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1833371989-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:23.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [451795381-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@769da30f at consistency null
2022-03-22 05:45:23.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [451795381-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:23.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123491660-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6da22d4f at consistency null
2022-03-22 05:45:23.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123491660-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:23.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1516276177-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c626692 at consistency null
2022-03-22 05:45:23.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1516276177-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:24.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [255340137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cff83af at consistency null
2022-03-22 05:45:24.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [255340137-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:24.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2070707786-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17e1cce4 at consistency null
2022-03-22 05:45:24.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2070707786-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:24.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180757474-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b555302 at consistency null
2022-03-22 05:45:24.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180757474-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:24.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1421217130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@749cfe0c at consistency null
2022-03-22 05:45:24.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1421217130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:24.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [439560947-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40ef9cb at consistency null
2022-03-22 05:45:24.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [439560947-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:24.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [45451729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d918342 at consistency null
2022-03-22 05:45:24.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [45451729-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:25.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [962032470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c9d1eb1 at consistency null
2022-03-22 05:45:25.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [962032470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:25.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1567548491-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fe2e7b0 at consistency null
2022-03-22 05:45:25.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1567548491-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:26.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1577662887-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cd18f26 at consistency null
2022-03-22 05:45:26.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1577662887-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:27.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893517494-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d4b806d at consistency null
2022-03-22 05:45:27.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893517494-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:27.163 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:45:32.191 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1913899064-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19976be3 at consistency null
2022-03-22 05:45:32.191 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1913899064-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:32.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484076685-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4110d006 at consistency null
2022-03-22 05:45:32.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484076685-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:32.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1336342565-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7188cfbd at consistency null
2022-03-22 05:45:32.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1336342565-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:32.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504251935-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@316109 at consistency null
2022-03-22 05:45:32.334 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504251935-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:32.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890720538-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@225bca9f at consistency null
2022-03-22 05:45:32.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890720538-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:32.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1143457899-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@650cbe32 at consistency null
2022-03-22 05:45:32.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1143457899-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:32.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1010329370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52ce7009 at consistency null
2022-03-22 05:45:32.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1010329370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:32.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [763408746-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59fc7a70 at consistency null
2022-03-22 05:45:32.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [763408746-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:32.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579906113-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@250f5ce2 at consistency null
2022-03-22 05:45:32.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579906113-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:33.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1477331688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49de9c0 at consistency null
2022-03-22 05:45:33.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1477331688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:34.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [661780036-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e883cd9 at consistency null
2022-03-22 05:45:34.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [661780036-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:35.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257038369-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@613f9fa6 at consistency null
2022-03-22 05:45:35.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257038369-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:35.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683112744-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5026c8ad at consistency null
2022-03-22 05:45:35.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683112744-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:35.846 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:45:40.741 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:45:40.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:45:40.858 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46411740-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b1ef880 at consistency null
2022-03-22 05:45:40.858 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [46411740-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:40.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [716126742-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@440f75ce at consistency null
2022-03-22 05:45:40.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [716126742-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:41.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899918962-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59500046 at consistency null
2022-03-22 05:45:41.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899918962-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:41.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [642706903-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f55c861 at consistency null
2022-03-22 05:45:41.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [642706903-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:41.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [98012173-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73c3edf3 at consistency null
2022-03-22 05:45:41.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [98012173-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:41.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1450928427-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cdd4740 at consistency null
2022-03-22 05:45:41.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1450928427-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:41.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123603882-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15563cad at consistency null
2022-03-22 05:45:41.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123603882-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:41.743 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795010554-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@681121ba at consistency null
2022-03-22 05:45:41.744 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795010554-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:42.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [384607231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ed09cbf at consistency null
2022-03-22 05:45:42.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [384607231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:42.557 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458469944-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ac7629c at consistency null
2022-03-22 05:45:42.557 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458469944-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:43.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2044661815-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@615823ba at consistency null
2022-03-22 05:45:43.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2044661815-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:43.365 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:45:48.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1093302211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25c2da at consistency null
2022-03-22 05:45:48.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1093302211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:48.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [841668418-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a5a9305 at consistency null
2022-03-22 05:45:48.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [841668418-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:48.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879653197-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bd8cb50 at consistency null
2022-03-22 05:45:48.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879653197-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:48.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [892436831-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b3d7e62 at consistency null
2022-03-22 05:45:48.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [892436831-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:48.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [666336341-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@152048cd at consistency null
2022-03-22 05:45:48.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [666336341-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:48.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136865062-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b7c23d7 at consistency null
2022-03-22 05:45:48.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136865062-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:49.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [73452042-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5623eba3 at consistency null
2022-03-22 05:45:49.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [73452042-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:49.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1038260384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bd516b at consistency null
2022-03-22 05:45:49.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1038260384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:49.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [556474013-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@114c0751 at consistency null
2022-03-22 05:45:49.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [556474013-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:49.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2068557697-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bc4fc34 at consistency null
2022-03-22 05:45:49.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2068557697-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:49.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046364280-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e295730 at consistency null
2022-03-22 05:45:49.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046364280-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:49.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1358408319-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f2ea52b at consistency null
2022-03-22 05:45:49.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1358408319-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:49.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [701602058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73d7d9a6 at consistency null
2022-03-22 05:45:49.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [701602058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:50.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264244246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a7d4a4e at consistency null
2022-03-22 05:45:50.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264244246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:50.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1046179164-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ef906e6 at consistency null
2022-03-22 05:45:50.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1046179164-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:51.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1402117306-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14a4f5ef at consistency null
2022-03-22 05:45:51.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1402117306-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:51.517 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:45:56.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982744075-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59f30434 at consistency null
2022-03-22 05:45:56.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982744075-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:56.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038514788-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@696c1a7e at consistency null
2022-03-22 05:45:56.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038514788-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:56.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475391877-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f24ff83 at consistency null
2022-03-22 05:45:56.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475391877-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:57.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1541791737-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c1c7405 at consistency null
2022-03-22 05:45:57.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1541791737-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:57.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1602208912-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d0f0916 at consistency null
2022-03-22 05:45:57.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1602208912-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:57.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [741214591-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14394f23 at consistency null
2022-03-22 05:45:57.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [741214591-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:57.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536132272-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ca972c0 at consistency null
2022-03-22 05:45:57.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536132272-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:57.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1780200608-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63aa990c at consistency null
2022-03-22 05:45:57.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1780200608-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:58.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273083447-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6087bd6 at consistency null
2022-03-22 05:45:58.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273083447-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:58.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391654341-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58966c63 at consistency null
2022-03-22 05:45:58.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391654341-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:58.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [767763108-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fbe3ac3 at consistency null
2022-03-22 05:45:58.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [767763108-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:59.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [441068808-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@786bb8c1 at consistency null
2022-03-22 05:45:59.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [441068808-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:45:59.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549561976-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3df8aad1 at consistency null
2022-03-22 05:45:59.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549561976-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:00.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [168590901-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56662c79 at consistency null
2022-03-22 05:46:00.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [168590901-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:00.509 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:46:05.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1055828946-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68306c54 at consistency null
2022-03-22 05:46:05.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1055828946-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:05.563 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1971839850-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4db87c23 at consistency null
2022-03-22 05:46:05.563 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1971839850-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:05.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256686098-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35eef967 at consistency null
2022-03-22 05:46:05.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256686098-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:05.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [594784429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39358a8e at consistency null
2022-03-22 05:46:05.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [594784429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:05.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1753755434-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37e2b0ef at consistency null
2022-03-22 05:46:05.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1753755434-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:06.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [576593378-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@750542b5 at consistency null
2022-03-22 05:46:06.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [576593378-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:07.164 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415572649-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47498464 at consistency null
2022-03-22 05:46:07.164 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415572649-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:07.987 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1788332119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52bd07f9 at consistency null
2022-03-22 05:46:07.987 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1788332119-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:08.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1137658704-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@707aa089 at consistency null
2022-03-22 05:46:08.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1137658704-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:08.805 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:46:10.769 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:46:10.773 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:46:13.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [288157638-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f3544d9 at consistency null
2022-03-22 05:46:13.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [288157638-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:13.872 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985490414-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24d49fff at consistency null
2022-03-22 05:46:13.873 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985490414-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:13.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1857864094-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1eecd653 at consistency null
2022-03-22 05:46:13.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1857864094-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:14.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469088466-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bb753ef at consistency null
2022-03-22 05:46:14.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469088466-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:14.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [557057337-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ea1dbcc at consistency null
2022-03-22 05:46:14.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [557057337-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:14.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1883742434-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e0a8080 at consistency null
2022-03-22 05:46:14.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1883742434-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:14.341 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029989701-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46ac975c at consistency null
2022-03-22 05:46:14.341 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029989701-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:14.480 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732607642-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47858c4d at consistency null
2022-03-22 05:46:14.480 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732607642-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:14.684 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [716599579-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@774a50ca at consistency null
2022-03-22 05:46:14.684 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [716599579-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:14.889 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1740346399-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d2ed023 at consistency null
2022-03-22 05:46:14.889 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1740346399-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:15.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1723796138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f9ab2ad at consistency null
2022-03-22 05:46:15.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1723796138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:15.297 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407858307-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5496a6f7 at consistency null
2022-03-22 05:46:15.298 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407858307-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:15.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23652408-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@717f60af at consistency null
2022-03-22 05:46:15.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23652408-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:16.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [977833830-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1abb096b at consistency null
2022-03-22 05:46:16.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [977833830-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:16.527 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:46:21.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607922399-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70dedf4b at consistency null
2022-03-22 05:46:21.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607922399-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:21.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1207202735-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@119e6eb9 at consistency null
2022-03-22 05:46:21.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1207202735-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:21.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18573889-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b6e1e3d at consistency null
2022-03-22 05:46:21.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18573889-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:21.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2144790203-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@595cbc6a at consistency null
2022-03-22 05:46:21.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2144790203-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:22.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [979981134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ffe048d at consistency null
2022-03-22 05:46:22.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [979981134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:22.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1583625895-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d77426d at consistency null
2022-03-22 05:46:22.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1583625895-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:22.244 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83301960-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56832c08 at consistency null
2022-03-22 05:46:22.244 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83301960-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:22.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281153875-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a00755e at consistency null
2022-03-22 05:46:22.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281153875-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:22.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1021252277-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4575ceff at consistency null
2022-03-22 05:46:22.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1021252277-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:22.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1131534983-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42969816 at consistency null
2022-03-22 05:46:22.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1131534983-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:23.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2131284471-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d3a1f20 at consistency null
2022-03-22 05:46:23.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2131284471-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:23.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436814086-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64f9315c at consistency null
2022-03-22 05:46:23.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436814086-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:23.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [296411581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f90090a at consistency null
2022-03-22 05:46:23.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [296411581-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:23.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1547875565-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@793076c6 at consistency null
2022-03-22 05:46:23.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1547875565-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:24.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1702642171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@242eb5df at consistency null
2022-03-22 05:46:24.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1702642171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:24.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276138601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9f013d7 at consistency null
2022-03-22 05:46:24.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276138601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:24.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1444165331-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a783ba at consistency null
2022-03-22 05:46:24.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1444165331-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:25.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523765024-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c5b43ee at consistency null
2022-03-22 05:46:25.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523765024-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:25.409 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:46:30.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [416087205-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d8a81f7 at consistency null
2022-03-22 05:46:30.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [416087205-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:30.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [618011415-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24ed413a at consistency null
2022-03-22 05:46:30.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [618011415-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:30.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401099248-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e6dac4e at consistency null
2022-03-22 05:46:30.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401099248-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:30.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214769061-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@388fd85e at consistency null
2022-03-22 05:46:30.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214769061-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:30.769 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1939599693-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41737381 at consistency null
2022-03-22 05:46:30.769 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1939599693-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:30.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728135818-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fd91e04 at consistency null
2022-03-22 05:46:30.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728135818-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:31.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752271625-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ca5dc6d at consistency null
2022-03-22 05:46:31.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752271625-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:31.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1209188611-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a2ea68f at consistency null
2022-03-22 05:46:31.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1209188611-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:31.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680818389-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@609b3530 at consistency null
2022-03-22 05:46:31.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680818389-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:32.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156715976-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27f52e19 at consistency null
2022-03-22 05:46:32.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156715976-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:33.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [328408418-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7561d83e at consistency null
2022-03-22 05:46:33.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [328408418-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:33.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1297603786-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a3402c3 at consistency null
2022-03-22 05:46:33.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1297603786-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:33.908 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:46:38.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1398801501-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bf9200 at consistency null
2022-03-22 05:46:38.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1398801501-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:38.970 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1624906994-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@db4f188 at consistency null
2022-03-22 05:46:38.970 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1624906994-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:39.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797875078-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76ef6828 at consistency null
2022-03-22 05:46:39.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797875078-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:39.139 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [418071973-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cd75e2c at consistency null
2022-03-22 05:46:39.139 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [418071973-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:39.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1266350089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1307ee24 at consistency null
2022-03-22 05:46:39.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1266350089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:39.470 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079801368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3915757d at consistency null
2022-03-22 05:46:39.471 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079801368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:39.677 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1532104629-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34b6e45f at consistency null
2022-03-22 05:46:39.678 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1532104629-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:39.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [425111319-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6246eadb at consistency null
2022-03-22 05:46:39.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [425111319-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:40.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [91272836-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e9c5e79 at consistency null
2022-03-22 05:46:40.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [91272836-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:40.539 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1153876402-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c30644a at consistency null
2022-03-22 05:46:40.540 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1153876402-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:40.778 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:46:40.783 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:46:41.375 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [253903160-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fa7f78f at consistency null
2022-03-22 05:46:41.375 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [253903160-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:41.377 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:46:46.389 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589326812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ae2ce92 at consistency null
2022-03-22 05:46:46.389 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589326812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:46.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [593768362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26a12621 at consistency null
2022-03-22 05:46:46.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [593768362-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:46.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373483301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79ec148b at consistency null
2022-03-22 05:46:46.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373483301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:46.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282085287-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19179fb6 at consistency null
2022-03-22 05:46:46.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282085287-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:46.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1822814079-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51d5458 at consistency null
2022-03-22 05:46:46.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1822814079-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:46.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1857979454-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a22faf4 at consistency null
2022-03-22 05:46:46.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1857979454-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:47.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1381385301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33f3fb77 at consistency null
2022-03-22 05:46:47.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1381385301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:47.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176801681-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16bc2890 at consistency null
2022-03-22 05:46:47.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176801681-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:47.548 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96030781-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@796c018b at consistency null
2022-03-22 05:46:47.549 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96030781-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:48.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532462528-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44a4a108 at consistency null
2022-03-22 05:46:48.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532462528-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:48.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2141728910-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68a0c2a8 at consistency null
2022-03-22 05:46:48.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2141728910-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:48.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493002483-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d6b5a3 at consistency null
2022-03-22 05:46:48.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493002483-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:49.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1385500122-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@671be890 at consistency null
2022-03-22 05:46:49.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1385500122-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:49.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680386291-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9c0c359 at consistency null
2022-03-22 05:46:49.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680386291-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:50.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371177260-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a703e5a at consistency null
2022-03-22 05:46:50.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371177260-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:50.329 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:46:55.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [520304154-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5938e51a at consistency null
2022-03-22 05:46:55.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [520304154-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:55.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1694635005-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@615fd9c5 at consistency null
2022-03-22 05:46:55.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1694635005-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:55.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [591279080-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e0a04bc at consistency null
2022-03-22 05:46:55.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [591279080-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:55.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223906932-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56375c9 at consistency null
2022-03-22 05:46:55.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223906932-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:55.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1313217174-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bed504a at consistency null
2022-03-22 05:46:55.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1313217174-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:56.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [992896344-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e32303 at consistency null
2022-03-22 05:46:56.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [992896344-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:56.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776136358-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a72e8c7 at consistency null
2022-03-22 05:46:56.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776136358-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:56.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1003424543-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24a10f7d at consistency null
2022-03-22 05:46:56.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1003424543-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:57.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1168323674-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@553f74a3 at consistency null
2022-03-22 05:46:57.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1168323674-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:59.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [233803027-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48d9be3c at consistency null
2022-03-22 05:46:59.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [233803027-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:46:59.305 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:47:04.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [154354824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65c05330 at consistency null
2022-03-22 05:47:04.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [154354824-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:04.390 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256661314-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49b66212 at consistency null
2022-03-22 05:47:04.390 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256661314-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:04.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372952480-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67828deb at consistency null
2022-03-22 05:47:04.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372952480-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:04.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1733053793-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@622ff779 at consistency null
2022-03-22 05:47:04.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1733053793-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:05.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959285252-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51161df at consistency null
2022-03-22 05:47:05.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959285252-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:05.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123113710-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fcd259d at consistency null
2022-03-22 05:47:05.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123113710-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:07.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1195448937-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c53025a at consistency null
2022-03-22 05:47:07.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1195448937-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:07.548 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:47:08.508 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:47:08.519 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-68, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:47:08.519 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-68, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:47:08.519 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-68, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:47:08.520 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-68, inFlight=0, closed=true] closing connection
2022-03-22 05:47:08.521 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 512000 milliseconds
2022-03-22 05:47:08.521 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 512000 ms
2022-03-22 05:47:08.522 DEBUG 10 --- [uster-worker-80] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:47:10.805 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:47:10.808 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:47:12.558 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1113809061-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@115ab896 at consistency null
2022-03-22 05:47:12.558 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1113809061-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:12.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1955418783-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f4b3cfb at consistency null
2022-03-22 05:47:12.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1955418783-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:12.669 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1198484736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@93c1342 at consistency null
2022-03-22 05:47:12.681 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1198484736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:12.793 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [137536982-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d6991ed at consistency null
2022-03-22 05:47:12.794 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [137536982-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:12.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1368234557-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@593314de at consistency null
2022-03-22 05:47:12.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1368234557-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:13.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1334453049-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fd957e7 at consistency null
2022-03-22 05:47:13.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1334453049-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:13.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1940996261-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7616f15d at consistency null
2022-03-22 05:47:13.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1940996261-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:13.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1857046288-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2721e08b at consistency null
2022-03-22 05:47:13.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1857046288-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:13.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406231689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66758555 at consistency null
2022-03-22 05:47:13.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406231689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:13.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1297880609-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33c68013 at consistency null
2022-03-22 05:47:13.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1297880609-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:13.946 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873579335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@729b132f at consistency null
2022-03-22 05:47:13.946 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873579335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:14.777 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484102162-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a16804f at consistency null
2022-03-22 05:47:14.777 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484102162-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:15.589 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5785785-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1083e975 at consistency null
2022-03-22 05:47:15.589 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5785785-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:15.591 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:47:20.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [756591501-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17c2356e at consistency null
2022-03-22 05:47:20.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [756591501-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:20.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388587154-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d6a4ed2 at consistency null
2022-03-22 05:47:20.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388587154-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:21.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1668871023-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5310a5c8 at consistency null
2022-03-22 05:47:21.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1668871023-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:21.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [556233273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a38012f at consistency null
2022-03-22 05:47:21.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [556233273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:21.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1233903955-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@354bd864 at consistency null
2022-03-22 05:47:21.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1233903955-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:21.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [336329534-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63a315e0 at consistency null
2022-03-22 05:47:21.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [336329534-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:22.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [818712923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a52cdc at consistency null
2022-03-22 05:47:22.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [818712923-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:22.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [719363009-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c8e745e at consistency null
2022-03-22 05:47:22.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [719363009-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:23.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1014462135-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c9a005b at consistency null
2022-03-22 05:47:23.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1014462135-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:24.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012133388-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23f60ed8 at consistency null
2022-03-22 05:47:24.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012133388-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:24.119 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:47:29.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603761123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c35b27c at consistency null
2022-03-22 05:47:29.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603761123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:29.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [225185998-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@537d6326 at consistency null
2022-03-22 05:47:29.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [225185998-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:29.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [178855194-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f3929ca at consistency null
2022-03-22 05:47:29.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [178855194-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:29.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [134605467-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1caa763b at consistency null
2022-03-22 05:47:29.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [134605467-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:29.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324415433-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ded9180 at consistency null
2022-03-22 05:47:29.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324415433-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:30.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [595028665-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4531a948 at consistency null
2022-03-22 05:47:30.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [595028665-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:30.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1635054080-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@300bf5d at consistency null
2022-03-22 05:47:30.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1635054080-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:31.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1182961031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cd41aa8 at consistency null
2022-03-22 05:47:31.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1182961031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:33.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [267717167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5da4e323 at consistency null
2022-03-22 05:47:33.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [267717167-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:33.028 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:47:38.054 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [399477706-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15977a36 at consistency null
2022-03-22 05:47:38.054 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [399477706-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1366138817-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13ebd58e at consistency null
2022-03-22 05:47:38.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1366138817-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64081630-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f069c01 at consistency null
2022-03-22 05:47:38.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64081630-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.277 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617748729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c1d539 at consistency null
2022-03-22 05:47:38.277 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617748729-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1328039538-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b29597d at consistency null
2022-03-22 05:47:38.346 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1328039538-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.456 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1921407937-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66a3ae33 at consistency null
2022-03-22 05:47:38.459 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1921407937-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2142035057-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d0ed072 at consistency null
2022-03-22 05:47:38.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2142035057-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267537581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@266da0f7 at consistency null
2022-03-22 05:47:38.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267537581-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.681 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176228472-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@476ef7f6 at consistency null
2022-03-22 05:47:38.682 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176228472-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109297961-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cc70b1e at consistency null
2022-03-22 05:47:38.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109297961-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:38.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [534108892-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6711d1b4 at consistency null
2022-03-22 05:47:38.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [534108892-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:39.083 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1020174325-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c30bfb9 at consistency null
2022-03-22 05:47:39.083 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1020174325-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:39.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [610140678-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c513c8e at consistency null
2022-03-22 05:47:39.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [610140678-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:39.558 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1020003629-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@184a5be1 at consistency null
2022-03-22 05:47:39.558 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1020003629-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:39.987 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [226376530-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71bfa52d at consistency null
2022-03-22 05:47:39.987 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [226376530-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:40.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1158097983-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@585b01d4 at consistency null
2022-03-22 05:47:40.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1158097983-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:40.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [295520427-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d7be31d at consistency null
2022-03-22 05:47:40.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [295520427-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:40.830 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:47:40.833 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:47:41.247 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294254808-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7242c00 at consistency null
2022-03-22 05:47:41.247 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294254808-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:41.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [838648965-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bbe6704 at consistency null
2022-03-22 05:47:41.692 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [838648965-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:41.700 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:47:46.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298559057-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@beb8206 at consistency null
2022-03-22 05:47:46.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298559057-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:46.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [910532934-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@191afb6d at consistency null
2022-03-22 05:47:46.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [910532934-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:46.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1562154836-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ff89062 at consistency null
2022-03-22 05:47:46.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1562154836-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:47.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [169060510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@470646de at consistency null
2022-03-22 05:47:47.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [169060510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:47.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1946789237-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@102a178a at consistency null
2022-03-22 05:47:47.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1946789237-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:47.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1868395811-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e254771 at consistency null
2022-03-22 05:47:47.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1868395811-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:47.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1447225771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2525a757 at consistency null
2022-03-22 05:47:47.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1447225771-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:48.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090714789-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50e0359b at consistency null
2022-03-22 05:47:48.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090714789-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:50.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2014956541-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1036ee4a at consistency null
2022-03-22 05:47:50.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2014956541-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:50.404 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:47:55.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2087275952-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64dc3df2 at consistency null
2022-03-22 05:47:55.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2087275952-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:55.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1034419239-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c4b2121 at consistency null
2022-03-22 05:47:55.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1034419239-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:55.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1805048034-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ab57080 at consistency null
2022-03-22 05:47:55.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1805048034-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:55.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1575399502-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f83fd89 at consistency null
2022-03-22 05:47:55.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1575399502-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:56.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809085071-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2af48571 at consistency null
2022-03-22 05:47:56.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809085071-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:56.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1762721046-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2345d58c at consistency null
2022-03-22 05:47:56.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1762721046-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:57.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [882669854-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12459014 at consistency null
2022-03-22 05:47:57.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [882669854-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:59.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552997123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12aa4860 at consistency null
2022-03-22 05:47:59.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552997123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:47:59.011 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:48:04.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [337156628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c7b8849 at consistency null
2022-03-22 05:48:04.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [337156628-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:04.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2056142284-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@119fc7ff at consistency null
2022-03-22 05:48:04.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2056142284-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:04.199 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [755891978-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f6420e8 at consistency null
2022-03-22 05:48:04.199 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [755891978-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:04.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2006937963-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f4d942c at consistency null
2022-03-22 05:48:04.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2006937963-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:04.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [654810550-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f34604c at consistency null
2022-03-22 05:48:04.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [654810550-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:05.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [753710279-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40fbfa6a at consistency null
2022-03-22 05:48:05.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [753710279-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:05.446 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851517951-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e16f1fd at consistency null
2022-03-22 05:48:05.446 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851517951-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:06.255 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1648614262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4df9f0e7 at consistency null
2022-03-22 05:48:06.255 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1648614262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:07.879 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2033324197-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4143df14 at consistency null
2022-03-22 05:48:07.879 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2033324197-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:07.883 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:48:10.837 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:48:10.845 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:48:12.900 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808569288-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ef455fc at consistency null
2022-03-22 05:48:12.900 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808569288-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:12.949 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [25479202-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2714cec6 at consistency null
2022-03-22 05:48:12.949 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [25479202-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:13.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103458575-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25686c0b at consistency null
2022-03-22 05:48:13.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103458575-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:13.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [348044094-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a8cbccd at consistency null
2022-03-22 05:48:13.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [348044094-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:13.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [721375208-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ab4dd5 at consistency null
2022-03-22 05:48:13.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [721375208-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:13.447 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220026701-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c8c138 at consistency null
2022-03-22 05:48:13.448 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220026701-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:13.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1910091170-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6126edfe at consistency null
2022-03-22 05:48:13.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1910091170-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:14.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [274698220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ed35fc3 at consistency null
2022-03-22 05:48:14.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [274698220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:14.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074729472-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22ca6029 at consistency null
2022-03-22 05:48:14.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074729472-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:15.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [659522163-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63caa449 at consistency null
2022-03-22 05:48:15.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [659522163-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:15.969 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [457678725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@519c90ed at consistency null
2022-03-22 05:48:15.969 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [457678725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:16.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1997231086-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@202669df at consistency null
2022-03-22 05:48:16.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1997231086-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:16.782 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:48:21.857 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1336725533-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1678797e at consistency null
2022-03-22 05:48:21.857 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1336725533-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:21.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [338403201-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cd1bc16 at consistency null
2022-03-22 05:48:21.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [338403201-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:22.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [867664182-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39ff5e99 at consistency null
2022-03-22 05:48:22.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [867664182-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:22.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504242317-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f628f5f at consistency null
2022-03-22 05:48:22.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504242317-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:22.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [923477865-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ed2bda9 at consistency null
2022-03-22 05:48:22.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [923477865-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:22.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2132046238-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d1a3f7c at consistency null
2022-03-22 05:48:22.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2132046238-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:23.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1889923451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3296dbd7 at consistency null
2022-03-22 05:48:23.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1889923451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:24.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [524336554-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59e5a2ad at consistency null
2022-03-22 05:48:24.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [524336554-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:24.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1906241860-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32b173cd at consistency null
2022-03-22 05:48:24.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1906241860-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:25.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769320552-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ebc5b38 at consistency null
2022-03-22 05:48:25.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769320552-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:25.826 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:48:30.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1923871873-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5af07b5b at consistency null
2022-03-22 05:48:30.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1923871873-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:31.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1755097747-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ef44a62 at consistency null
2022-03-22 05:48:31.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1755097747-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:31.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1846436373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17e26b6a at consistency null
2022-03-22 05:48:31.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1846436373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:31.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345845786-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b3b8735 at consistency null
2022-03-22 05:48:31.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345845786-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:31.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [960483828-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72c80baa at consistency null
2022-03-22 05:48:31.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [960483828-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:31.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [880422002-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e63e8cb at consistency null
2022-03-22 05:48:31.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [880422002-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:32.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [704245282-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a5e7556 at consistency null
2022-03-22 05:48:32.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [704245282-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:32.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1419319015-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@dab5707 at consistency null
2022-03-22 05:48:32.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1419319015-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:33.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [334421489-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f8c8843 at consistency null
2022-03-22 05:48:33.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [334421489-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:33.244 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:48:38.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989306946-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e481ac9 at consistency null
2022-03-22 05:48:38.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989306946-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:38.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284951116-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57353093 at consistency null
2022-03-22 05:48:38.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284951116-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:38.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [206438554-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f1461d at consistency null
2022-03-22 05:48:38.356 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [206438554-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:38.391 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1772766647-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b6408d0 at consistency null
2022-03-22 05:48:38.391 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1772766647-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:38.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [202371770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4564df27 at consistency null
2022-03-22 05:48:38.438 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [202371770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:38.470 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2081521429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b2bceb0 at consistency null
2022-03-22 05:48:38.470 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2081521429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:38.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [11008733-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31e0b666 at consistency null
2022-03-22 05:48:38.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [11008733-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:38.633 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [991658115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a390a02 at consistency null
2022-03-22 05:48:38.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [991658115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:38.866 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223835512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29c77985 at consistency null
2022-03-22 05:48:38.867 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223835512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:39.283 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1064044708-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c7c6c43 at consistency null
2022-03-22 05:48:39.283 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1064044708-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:39.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2256006-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@789ba02c at consistency null
2022-03-22 05:48:39.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2256006-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:40.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742400409-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a70dc27 at consistency null
2022-03-22 05:48:40.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742400409-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:40.566 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1040843237-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@686999bd at consistency null
2022-03-22 05:48:40.566 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1040843237-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:40.865 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:48:40.880 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:48:41.388 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1635518376-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4758ee87 at consistency null
2022-03-22 05:48:41.388 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1635518376-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:42.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1915294045-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6267ac80 at consistency null
2022-03-22 05:48:42.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1915294045-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:42.210 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:48:47.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1117764258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e8ad50f at consistency null
2022-03-22 05:48:47.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1117764258-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:47.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754228797-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a8a0312 at consistency null
2022-03-22 05:48:47.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754228797-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:47.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [758422860-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3aa6ee12 at consistency null
2022-03-22 05:48:47.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [758422860-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:47.626 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397267534-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65fa566a at consistency null
2022-03-22 05:48:47.626 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397267534-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:48.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [676864179-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47e8d144 at consistency null
2022-03-22 05:48:48.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [676864179-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:48.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550852339-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ab371d5 at consistency null
2022-03-22 05:48:48.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550852339-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:49.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [688021848-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dcd27ef at consistency null
2022-03-22 05:48:49.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [688021848-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:50.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355082132-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@565dfe5c at consistency null
2022-03-22 05:48:50.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355082132-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:50.944 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:48:56.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [963643015-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@573e7722 at consistency null
2022-03-22 05:48:56.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [963643015-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:56.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [857259412-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65289c9 at consistency null
2022-03-22 05:48:56.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [857259412-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:56.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [162136590-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@123434b3 at consistency null
2022-03-22 05:48:56.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [162136590-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:56.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2119549670-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ba287b7 at consistency null
2022-03-22 05:48:56.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2119549670-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:56.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1193850773-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f016286 at consistency null
2022-03-22 05:48:56.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1193850773-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:56.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [833246969-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d0fbe6e at consistency null
2022-03-22 05:48:56.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [833246969-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:56.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1749653680-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e2d80d0 at consistency null
2022-03-22 05:48:56.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1749653680-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:56.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [953886125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@118d359d at consistency null
2022-03-22 05:48:56.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [953886125-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:57.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [978172220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@159c6078 at consistency null
2022-03-22 05:48:57.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [978172220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:57.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1496526098-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7500406b at consistency null
2022-03-22 05:48:57.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1496526098-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:57.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2107351708-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37e9751b at consistency null
2022-03-22 05:48:57.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2107351708-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:57.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [470550223-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f308fa6 at consistency null
2022-03-22 05:48:57.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [470550223-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:58.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1264568363-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39d9a998 at consistency null
2022-03-22 05:48:58.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1264568363-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:58.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [189016047-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e2da5d0 at consistency null
2022-03-22 05:48:58.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [189016047-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:59.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [736881497-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@321040a3 at consistency null
2022-03-22 05:48:59.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [736881497-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:48:59.309 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:49:04.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1462610581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7916b4f8 at consistency null
2022-03-22 05:49:04.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1462610581-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:04.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1652965067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@df7472b at consistency null
2022-03-22 05:49:04.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1652965067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:04.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1222055467-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ed76060 at consistency null
2022-03-22 05:49:04.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1222055467-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:04.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1007848779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e722841 at consistency null
2022-03-22 05:49:04.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1007848779-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:04.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [114422110-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@520ffe03 at consistency null
2022-03-22 05:49:04.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [114422110-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:05.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1441986345-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65762576 at consistency null
2022-03-22 05:49:05.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1441986345-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:05.744 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [627361266-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70378dd8 at consistency null
2022-03-22 05:49:05.744 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [627361266-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:06.574 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229928128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f0893bf at consistency null
2022-03-22 05:49:06.575 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229928128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:08.188 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82192169-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52da85bb at consistency null
2022-03-22 05:49:08.188 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82192169-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:08.196 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:49:10.881 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:49:10.892 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:49:13.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1190118179-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d34703 at consistency null
2022-03-22 05:49:13.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1190118179-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:13.263 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1168622937-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a46534c at consistency null
2022-03-22 05:49:13.263 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1168622937-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:13.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801151297-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71e8b4d0 at consistency null
2022-03-22 05:49:13.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801151297-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:13.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1580322412-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7799000b at consistency null
2022-03-22 05:49:13.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1580322412-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:13.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1719658578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6beebaa4 at consistency null
2022-03-22 05:49:13.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1719658578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:13.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487158985-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13ca48a6 at consistency null
2022-03-22 05:49:13.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487158985-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:13.863 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [762862914-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2730303b at consistency null
2022-03-22 05:49:13.863 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [762862914-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:14.272 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2112533551-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1816e58e at consistency null
2022-03-22 05:49:14.272 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2112533551-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:15.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1691283128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b347b2 at consistency null
2022-03-22 05:49:15.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1691283128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:16.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [842490324-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@477da0ba at consistency null
2022-03-22 05:49:16.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [842490324-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:16.742 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:49:21.834 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1565173917-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61eae186 at consistency null
2022-03-22 05:49:21.834 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1565173917-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:21.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1974019287-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e7f52a0 at consistency null
2022-03-22 05:49:21.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1974019287-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:22.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1719775220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52ed020a at consistency null
2022-03-22 05:49:22.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1719775220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:22.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1873894193-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78bf009d at consistency null
2022-03-22 05:49:22.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1873894193-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:22.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1740052506-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4eef38dd at consistency null
2022-03-22 05:49:22.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1740052506-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:22.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535379367-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ab199f2 at consistency null
2022-03-22 05:49:22.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535379367-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:22.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [806481196-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@407639fc at consistency null
2022-03-22 05:49:22.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [806481196-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:22.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436303564-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@404d6287 at consistency null
2022-03-22 05:49:22.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436303564-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:22.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171242925-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77d38383 at consistency null
2022-03-22 05:49:22.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171242925-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:22.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903583859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15bfa82 at consistency null
2022-03-22 05:49:22.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903583859-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:23.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1311873532-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70138a8c at consistency null
2022-03-22 05:49:23.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1311873532-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:23.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2031889110-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5be18753 at consistency null
2022-03-22 05:49:23.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2031889110-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:23.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420308933-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30c5c837 at consistency null
2022-03-22 05:49:23.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420308933-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:23.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [370128089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e14ab47 at consistency null
2022-03-22 05:49:23.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [370128089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:24.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1412641181-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bf0be3 at consistency null
2022-03-22 05:49:24.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1412641181-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:24.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281489751-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55b3dff8 at consistency null
2022-03-22 05:49:24.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281489751-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:25.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506991367-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b292109 at consistency null
2022-03-22 05:49:25.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506991367-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:25.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839910296-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@acf5b8 at consistency null
2022-03-22 05:49:25.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839910296-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:25.613 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:49:30.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2145543070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a36165d at consistency null
2022-03-22 05:49:30.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2145543070-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:30.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1016763510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31b15afc at consistency null
2022-03-22 05:49:30.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1016763510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:30.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936677895-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@339214aa at consistency null
2022-03-22 05:49:30.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936677895-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:30.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1104256305-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54742655 at consistency null
2022-03-22 05:49:30.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1104256305-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:30.962 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [759425030-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1eba4678 at consistency null
2022-03-22 05:49:30.962 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [759425030-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:31.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1158865148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@256789e9 at consistency null
2022-03-22 05:49:31.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1158865148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:31.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1284772669-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f3feab at consistency null
2022-03-22 05:49:31.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1284772669-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:31.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [31532939-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73c026ae at consistency null
2022-03-22 05:49:31.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [31532939-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:31.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [860941417-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f053728 at consistency null
2022-03-22 05:49:31.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [860941417-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:31.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1628962815-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2634c9e0 at consistency null
2022-03-22 05:49:31.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1628962815-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:31.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [225838363-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f2b7d2b at consistency null
2022-03-22 05:49:31.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [225838363-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:32.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [86222401-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d2cc936 at consistency null
2022-03-22 05:49:32.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [86222401-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:33.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [832839976-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b060bcc at consistency null
2022-03-22 05:49:33.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [832839976-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:33.132 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:49:38.171 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871890660-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b7da285 at consistency null
2022-03-22 05:49:38.171 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871890660-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:38.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383159504-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f5b5ee4 at consistency null
2022-03-22 05:49:38.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383159504-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:38.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1257825137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14154540 at consistency null
2022-03-22 05:49:38.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1257825137-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:38.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1514485351-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fa67465 at consistency null
2022-03-22 05:49:38.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1514485351-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:38.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [602476579-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49f9889e at consistency null
2022-03-22 05:49:38.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [602476579-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:38.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079724888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cb0433b at consistency null
2022-03-22 05:49:38.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079724888-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:38.744 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981321788-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12313ca8 at consistency null
2022-03-22 05:49:38.744 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981321788-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:38.953 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [85812374-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ef877d7 at consistency null
2022-03-22 05:49:38.954 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [85812374-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:39.178 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1855853598-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b40b9ee at consistency null
2022-03-22 05:49:39.179 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1855853598-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:39.392 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621383744-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75478026 at consistency null
2022-03-22 05:49:39.392 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621383744-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:39.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389185455-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e88365a at consistency null
2022-03-22 05:49:39.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389185455-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:40.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2036699147-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7353a293 at consistency null
2022-03-22 05:49:40.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2036699147-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:40.636 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:49:40.909 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:49:40.919 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:49:45.647 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1740981725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d82eade at consistency null
2022-03-22 05:49:45.647 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1740981725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:45.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245842677-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@640d19f8 at consistency null
2022-03-22 05:49:45.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245842677-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:45.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [183274377-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b98d9c5 at consistency null
2022-03-22 05:49:45.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [183274377-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:46.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599911703-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2531ad1f at consistency null
2022-03-22 05:49:46.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599911703-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:46.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310045324-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fdf777d at consistency null
2022-03-22 05:49:46.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310045324-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:47.248 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [654226140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13917fa7 at consistency null
2022-03-22 05:49:47.248 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [654226140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:48.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1218495817-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e4adae1 at consistency null
2022-03-22 05:49:48.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1218495817-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:48.908 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:49:54.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1920874309-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@363bc562 at consistency null
2022-03-22 05:49:54.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1920874309-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:54.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [932547852-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@523b2847 at consistency null
2022-03-22 05:49:54.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [932547852-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:54.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680781680-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4acf19e0 at consistency null
2022-03-22 05:49:54.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680781680-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:54.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [711633457-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29a5f8a at consistency null
2022-03-22 05:49:54.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [711633457-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:54.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [699233318-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ae28a6e at consistency null
2022-03-22 05:49:54.544 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [699233318-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:54.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [852189682-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6190fe65 at consistency null
2022-03-22 05:49:54.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [852189682-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:54.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603972586-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53a2fbae at consistency null
2022-03-22 05:49:54.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603972586-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:55.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555824262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@394885f at consistency null
2022-03-22 05:49:55.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555824262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:55.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [519421299-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ddbd3b9 at consistency null
2022-03-22 05:49:55.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [519421299-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:55.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2010680238-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bde69b1 at consistency null
2022-03-22 05:49:55.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2010680238-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:56.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [736413196-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f659eef at consistency null
2022-03-22 05:49:56.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [736413196-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:57.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [673727032-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7894232c at consistency null
2022-03-22 05:49:57.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [673727032-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:57.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461728875-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b71619c at consistency null
2022-03-22 05:49:57.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461728875-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:49:57.828 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:50:02.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674435406-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@330a18a2 at consistency null
2022-03-22 05:50:02.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674435406-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:03.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636118754-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@406b7aca at consistency null
2022-03-22 05:50:03.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636118754-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:03.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1329191397-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19bfbca8 at consistency null
2022-03-22 05:50:03.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1329191397-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:03.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146374237-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49124382 at consistency null
2022-03-22 05:50:03.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146374237-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:03.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340528015-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a3e9d0a at consistency null
2022-03-22 05:50:03.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340528015-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:03.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961378279-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31980fad at consistency null
2022-03-22 05:50:03.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961378279-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:03.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [951744474-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5526afca at consistency null
2022-03-22 05:50:03.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [951744474-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:03.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1061222250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e73e8bf at consistency null
2022-03-22 05:50:03.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1061222250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:04.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181943712-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@507be2b6 at consistency null
2022-03-22 05:50:04.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181943712-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:05.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [349414086-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@216f6a73 at consistency null
2022-03-22 05:50:05.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [349414086-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:06.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2134643489-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41d7d96b at consistency null
2022-03-22 05:50:06.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2134643489-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:06.041 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:50:10.921 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:50:10.924 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:50:11.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1199521803-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50f0f7fe at consistency null
2022-03-22 05:50:11.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1199521803-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:11.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [426709315-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f11c104 at consistency null
2022-03-22 05:50:11.091 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [426709315-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:11.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [279139642-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f71467 at consistency null
2022-03-22 05:50:11.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [279139642-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:11.269 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [261749012-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a148933 at consistency null
2022-03-22 05:50:11.285 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [261749012-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:11.489 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [853254132-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d8cde88 at consistency null
2022-03-22 05:50:11.489 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [853254132-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:11.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407153637-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cc423dc at consistency null
2022-03-22 05:50:11.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407153637-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:11.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499455521-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@412d540f at consistency null
2022-03-22 05:50:11.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499455521-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:12.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [432988999-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f2e0008 at consistency null
2022-03-22 05:50:12.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [432988999-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:13.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1996811189-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@541df5fb at consistency null
2022-03-22 05:50:13.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1996811189-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:14.764 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599916269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@99af667 at consistency null
2022-03-22 05:50:14.764 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599916269-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:14.766 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:50:19.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040821199-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22bc4677 at consistency null
2022-03-22 05:50:19.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040821199-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:19.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396256615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d7d2c68 at consistency null
2022-03-22 05:50:19.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396256615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:20.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1201110370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76f4aa11 at consistency null
2022-03-22 05:50:20.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1201110370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:20.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117455581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28835366 at consistency null
2022-03-22 05:50:20.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117455581-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:20.441 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619042521-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68cc37ff at consistency null
2022-03-22 05:50:20.441 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619042521-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:20.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1085553279-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cf648e2 at consistency null
2022-03-22 05:50:20.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1085553279-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:20.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1010053990-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f01c74f at consistency null
2022-03-22 05:50:20.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1010053990-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:21.239 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1814515995-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2dabbfaa at consistency null
2022-03-22 05:50:21.239 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1814515995-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:21.672 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1719538008-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24d1162f at consistency null
2022-03-22 05:50:21.672 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1719538008-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:22.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [125351978-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a9eeb0f at consistency null
2022-03-22 05:50:22.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [125351978-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:23.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [441062560-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bb464ac at consistency null
2022-03-22 05:50:23.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [441062560-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:23.209 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:50:28.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286807931-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b12d49f at consistency null
2022-03-22 05:50:28.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286807931-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:28.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523653988-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@175ecdbc at consistency null
2022-03-22 05:50:28.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523653988-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:28.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [413812732-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a358acb at consistency null
2022-03-22 05:50:28.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [413812732-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:28.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1385148253-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@524c1825 at consistency null
2022-03-22 05:50:28.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1385148253-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:28.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123921550-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fd798df at consistency null
2022-03-22 05:50:28.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123921550-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:29.035 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1435165499-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e8599b4 at consistency null
2022-03-22 05:50:29.035 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1435165499-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:29.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [534105041-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8703b7d at consistency null
2022-03-22 05:50:29.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [534105041-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:30.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [806320689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a4f8766 at consistency null
2022-03-22 05:50:30.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [806320689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:32.135 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542748202-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@791de190 at consistency null
2022-03-22 05:50:32.135 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542748202-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:32.136 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:50:37.238 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575144481-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4be78f3e at consistency null
2022-03-22 05:50:37.238 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575144481-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:37.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [326295290-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42c5feec at consistency null
2022-03-22 05:50:37.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [326295290-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:37.365 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1491595351-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4772565 at consistency null
2022-03-22 05:50:37.365 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1491595351-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:37.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [891432351-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1270551a at consistency null
2022-03-22 05:50:37.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [891432351-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:37.683 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469671440-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6599b615 at consistency null
2022-03-22 05:50:37.683 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469671440-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:37.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1385818391-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@84616d4 at consistency null
2022-03-22 05:50:37.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1385818391-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:38.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899407451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35a756b4 at consistency null
2022-03-22 05:50:38.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899407451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:38.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [292639650-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25ef0d9f at consistency null
2022-03-22 05:50:38.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [292639650-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:39.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [221536372-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d6e2e29 at consistency null
2022-03-22 05:50:39.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [221536372-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:40.099 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284750228-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@352669e6 at consistency null
2022-03-22 05:50:40.100 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284750228-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:40.948 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2080017622-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a3b9b00 at consistency null
2022-03-22 05:50:40.948 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2080017622-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:40.958 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:50:40.962 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:50:40.980 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:50:46.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1619868744-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6765fa2d at consistency null
2022-03-22 05:50:46.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1619868744-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:46.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318373357-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b068f8e at consistency null
2022-03-22 05:50:46.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318373357-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:46.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431483898-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1de9bc51 at consistency null
2022-03-22 05:50:46.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431483898-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:46.396 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906250278-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b6cd777 at consistency null
2022-03-22 05:50:46.396 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906250278-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:46.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063411154-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3390eea5 at consistency null
2022-03-22 05:50:46.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063411154-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:46.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1461804668-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cc57da6 at consistency null
2022-03-22 05:50:46.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1461804668-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:47.142 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903138649-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c1e515b at consistency null
2022-03-22 05:50:47.142 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903138649-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:47.382 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1698604074-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5154e0c0 at consistency null
2022-03-22 05:50:47.382 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1698604074-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:47.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1218439570-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6585d989 at consistency null
2022-03-22 05:50:47.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1218439570-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:48.265 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361794627-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@521aa0d2 at consistency null
2022-03-22 05:50:48.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361794627-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:49.083 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407377143-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c30b743 at consistency null
2022-03-22 05:50:49.083 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407377143-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:49.088 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:50:54.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147319840-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a1418e5 at consistency null
2022-03-22 05:50:54.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147319840-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:54.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067665363-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5196d3bf at consistency null
2022-03-22 05:50:54.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067665363-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:54.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [775960304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@560df56f at consistency null
2022-03-22 05:50:54.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [775960304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:54.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1987534947-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3feeca42 at consistency null
2022-03-22 05:50:54.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1987534947-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:54.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1105601633-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5073e6a at consistency null
2022-03-22 05:50:54.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1105601633-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:55.049 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692298833-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62eb0e5e at consistency null
2022-03-22 05:50:55.053 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692298833-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:55.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267086220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bf429e1 at consistency null
2022-03-22 05:50:55.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267086220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:55.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [104806119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12815407 at consistency null
2022-03-22 05:50:55.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [104806119-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:56.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1988808557-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ec9cc53 at consistency null
2022-03-22 05:50:56.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1988808557-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:57.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630640907-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@146abe2a at consistency null
2022-03-22 05:50:57.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1630640907-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:50:57.710 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:51:02.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458346226-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6eab3667 at consistency null
2022-03-22 05:51:02.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458346226-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:02.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1688623985-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@769266d9 at consistency null
2022-03-22 05:51:02.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1688623985-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:02.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1157042479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5783f013 at consistency null
2022-03-22 05:51:02.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1157042479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:02.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695415673-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@602cfc8a at consistency null
2022-03-22 05:51:02.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695415673-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:03.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1010847483-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5edfaf8 at consistency null
2022-03-22 05:51:03.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1010847483-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:03.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1793845201-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fd6fcae at consistency null
2022-03-22 05:51:03.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1793845201-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:03.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [376040906-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@171d8c2f at consistency null
2022-03-22 05:51:03.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [376040906-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:04.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [640810117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10d2eaea at consistency null
2022-03-22 05:51:04.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [640810117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:04.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [406341934-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a82e832 at consistency null
2022-03-22 05:51:04.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [406341934-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:05.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [760799383-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12d95840 at consistency null
2022-03-22 05:51:05.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [760799383-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:06.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1946574667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@695ed2c0 at consistency null
2022-03-22 05:51:06.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1946574667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:06.349 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:51:10.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:51:11.006 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:51:11.373 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576685722-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e5cf0bb at consistency null
2022-03-22 05:51:11.373 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576685722-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:11.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [131142450-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f2463a at consistency null
2022-03-22 05:51:11.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [131142450-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:11.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835472647-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b31055c at consistency null
2022-03-22 05:51:11.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835472647-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:11.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [735965671-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44e60428 at consistency null
2022-03-22 05:51:11.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [735965671-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:11.767 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [606870150-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4db6ed27 at consistency null
2022-03-22 05:51:11.767 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [606870150-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:11.996 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [736875954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ce4b13d at consistency null
2022-03-22 05:51:11.996 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [736875954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:12.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128687335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fac5418 at consistency null
2022-03-22 05:51:12.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128687335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:12.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [989307925-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b9fc681 at consistency null
2022-03-22 05:51:12.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [989307925-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:13.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2118873575-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bac300 at consistency null
2022-03-22 05:51:13.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2118873575-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:13.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [200834521-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@782ee753 at consistency null
2022-03-22 05:51:13.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [200834521-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:14.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095645257-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15cf70da at consistency null
2022-03-22 05:51:14.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095645257-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:14.749 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:51:19.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2075772621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10e1072c at consistency null
2022-03-22 05:51:19.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2075772621-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:19.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1582842900-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@471464f3 at consistency null
2022-03-22 05:51:19.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1582842900-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:20.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [369552563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@480f073d at consistency null
2022-03-22 05:51:20.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [369552563-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:20.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575233904-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26a8e746 at consistency null
2022-03-22 05:51:20.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575233904-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:20.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [393439834-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e684cd8 at consistency null
2022-03-22 05:51:20.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [393439834-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:20.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [194507844-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5abf941a at consistency null
2022-03-22 05:51:20.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [194507844-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:20.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718664433-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ab9469f at consistency null
2022-03-22 05:51:20.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718664433-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:21.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1116703236-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bca0e59 at consistency null
2022-03-22 05:51:21.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1116703236-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:21.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1310857653-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77389d74 at consistency null
2022-03-22 05:51:21.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1310857653-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:22.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049171937-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5181c8cd at consistency null
2022-03-22 05:51:22.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049171937-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:22.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2137310766-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71ace1dc at consistency null
2022-03-22 05:51:22.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2137310766-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:23.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099279468-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2acf9176 at consistency null
2022-03-22 05:51:23.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099279468-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:23.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [531950168-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19af6a66 at consistency null
2022-03-22 05:51:23.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [531950168-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:23.619 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:51:28.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2055023782-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46b70e3e at consistency null
2022-03-22 05:51:28.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2055023782-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:28.754 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603050097-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5003b535 at consistency null
2022-03-22 05:51:28.754 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603050097-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:28.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341914799-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@691ca62 at consistency null
2022-03-22 05:51:28.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341914799-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:28.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2007313186-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@640f6c8c at consistency null
2022-03-22 05:51:28.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2007313186-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:29.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1689133420-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ab758cd at consistency null
2022-03-22 05:51:29.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1689133420-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:29.060 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1232144336-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57d0ecb3 at consistency null
2022-03-22 05:51:29.060 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1232144336-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:29.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1500485335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f9aeef1 at consistency null
2022-03-22 05:51:29.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1500485335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:29.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [130894563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40ee8282 at consistency null
2022-03-22 05:51:29.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [130894563-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:29.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310230583-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@603fcb62 at consistency null
2022-03-22 05:51:29.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310230583-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:30.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1400071215-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30da16a2 at consistency null
2022-03-22 05:51:30.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1400071215-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:30.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1980729784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38bf8b2e at consistency null
2022-03-22 05:51:30.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1980729784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:31.041 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [460869576-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c71e429 at consistency null
2022-03-22 05:51:31.041 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [460869576-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:31.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1686886038-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@120b5a1e at consistency null
2022-03-22 05:51:31.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1686886038-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:31.917 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:51:37.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [747734068-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@720431fb at consistency null
2022-03-22 05:51:37.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [747734068-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:37.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [651795031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3626226e at consistency null
2022-03-22 05:51:37.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [651795031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:37.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2139522891-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35f5c82f at consistency null
2022-03-22 05:51:37.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2139522891-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:37.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1157873995-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d66bf25 at consistency null
2022-03-22 05:51:37.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1157873995-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:37.435 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1704156667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ebed29b at consistency null
2022-03-22 05:51:37.435 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1704156667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:37.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [267548732-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@290cb070 at consistency null
2022-03-22 05:51:37.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [267548732-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:37.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1149573718-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3223a1ce at consistency null
2022-03-22 05:51:37.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1149573718-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:38.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [534221332-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7af4af38 at consistency null
2022-03-22 05:51:38.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [534221332-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:38.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1616148624-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@220680f7 at consistency null
2022-03-22 05:51:38.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1616148624-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:39.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2096881470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e452bf7 at consistency null
2022-03-22 05:51:39.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2096881470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:40.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [272645469-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4147da20 at consistency null
2022-03-22 05:51:40.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [272645469-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:40.355 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:51:41.016 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:51:41.033 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:51:45.372 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [858016371-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e15305f at consistency null
2022-03-22 05:51:45.372 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [858016371-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:45.433 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886856319-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8821a52 at consistency null
2022-03-22 05:51:45.433 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886856319-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:45.499 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1381439328-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fdf9226 at consistency null
2022-03-22 05:51:45.499 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1381439328-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:45.558 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1139569314-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3036d727 at consistency null
2022-03-22 05:51:45.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1139569314-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:45.665 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1384828798-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7968eb6 at consistency null
2022-03-22 05:51:45.665 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1384828798-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:45.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [462904716-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fde549c at consistency null
2022-03-22 05:51:45.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [462904716-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:45.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1896985993-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a2aceaf at consistency null
2022-03-22 05:51:45.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1896985993-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:46.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1372106859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@197a1662 at consistency null
2022-03-22 05:51:46.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1372106859-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:46.178 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1438479379-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74edcb65 at consistency null
2022-03-22 05:51:46.178 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1438479379-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:46.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1961306133-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45a09ab8 at consistency null
2022-03-22 05:51:46.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1961306133-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:46.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136738557-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39836b15 at consistency null
2022-03-22 05:51:46.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136738557-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:46.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [477742202-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7003b341 at consistency null
2022-03-22 05:51:46.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [477742202-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:47.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [576333322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6aaff8c7 at consistency null
2022-03-22 05:51:47.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [576333322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:47.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1865422292-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b1bca70 at consistency null
2022-03-22 05:51:47.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1865422292-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:48.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1410996049-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6416e413 at consistency null
2022-03-22 05:51:48.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1410996049-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:48.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [101860284-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71ea4196 at consistency null
2022-03-22 05:51:48.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [101860284-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:48.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1963675394-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4575d91 at consistency null
2022-03-22 05:51:48.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1963675394-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:49.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [444234434-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b147e21 at consistency null
2022-03-22 05:51:49.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [444234434-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:49.322 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:51:54.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [517883459-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48f1272c at consistency null
2022-03-22 05:51:54.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [517883459-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:54.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180742535-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bb43955 at consistency null
2022-03-22 05:51:54.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180742535-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:54.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622445698-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55b14a67 at consistency null
2022-03-22 05:51:54.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622445698-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:54.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820071561-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d6cd386 at consistency null
2022-03-22 05:51:54.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820071561-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:54.968 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [8169755-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c655140 at consistency null
2022-03-22 05:51:54.968 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [8169755-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:55.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [634446146-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6651ab08 at consistency null
2022-03-22 05:51:55.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [634446146-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:55.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1317412014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@396c27d7 at consistency null
2022-03-22 05:51:55.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1317412014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:55.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1120048041-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3297339f at consistency null
2022-03-22 05:51:55.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1120048041-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:55.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [664002203-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34fe8bdc at consistency null
2022-03-22 05:51:55.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [664002203-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:55.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [90960346-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f9c7fa0 at consistency null
2022-03-22 05:51:55.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [90960346-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:55.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079784629-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@796a91b at consistency null
2022-03-22 05:51:55.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079784629-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:56.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [771712441-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c1679bd at consistency null
2022-03-22 05:51:56.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [771712441-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:56.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [337711515-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cf6b388 at consistency null
2022-03-22 05:51:56.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [337711515-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:56.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1319032804-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70aba4c9 at consistency null
2022-03-22 05:51:56.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1319032804-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:57.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47287252-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@424705b6 at consistency null
2022-03-22 05:51:57.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47287252-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:51:57.830 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:52:02.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1414919322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d2347d2 at consistency null
2022-03-22 05:52:02.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1414919322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:03.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1222941250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b0c10d at consistency null
2022-03-22 05:52:03.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1222941250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:03.067 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887257933-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2910a0be at consistency null
2022-03-22 05:52:03.068 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887257933-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:03.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2122121149-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53007a9f at consistency null
2022-03-22 05:52:03.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2122121149-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:03.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [91629004-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34fabafa at consistency null
2022-03-22 05:52:03.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [91629004-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:03.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1764526506-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36c0bdc1 at consistency null
2022-03-22 05:52:03.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1764526506-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:03.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1935797999-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a79fef7 at consistency null
2022-03-22 05:52:03.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1935797999-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:04.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1786340905-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4afdbba5 at consistency null
2022-03-22 05:52:04.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1786340905-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:04.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578465007-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bb1e56a at consistency null
2022-03-22 05:52:04.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578465007-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:04.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1408877918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18ac6b68 at consistency null
2022-03-22 05:52:04.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1408877918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:04.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789798744-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d77b0cc at consistency null
2022-03-22 05:52:04.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789798744-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:05.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [251280573-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a701a54 at consistency null
2022-03-22 05:52:05.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [251280573-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:06.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2074968451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ece28c5 at consistency null
2022-03-22 05:52:06.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2074968451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:06.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1717790877-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35c2ed5f at consistency null
2022-03-22 05:52:06.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1717790877-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:06.841 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:52:11.046 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:52:11.049 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:52:11.846 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396642454-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4543cafb at consistency null
2022-03-22 05:52:11.846 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396642454-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:11.886 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [209699736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f8bdabd at consistency null
2022-03-22 05:52:11.886 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [209699736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:11.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466217485-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10bc5789 at consistency null
2022-03-22 05:52:11.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466217485-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:12.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1720933993-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3236f8d4 at consistency null
2022-03-22 05:52:12.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1720933993-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:12.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1293110159-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6979e87b at consistency null
2022-03-22 05:52:12.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1293110159-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:12.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1057108662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@185699db at consistency null
2022-03-22 05:52:12.461 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1057108662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:12.666 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1800355545-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a249547 at consistency null
2022-03-22 05:52:12.666 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1800355545-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:13.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391333327-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fd1208f at consistency null
2022-03-22 05:52:13.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391333327-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:13.488 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1078672281-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49cd373e at consistency null
2022-03-22 05:52:13.489 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1078672281-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:14.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1852974756-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6579f79d at consistency null
2022-03-22 05:52:14.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1852974756-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:14.317 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:52:19.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2094625861-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d8a4068 at consistency null
2022-03-22 05:52:19.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2094625861-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:19.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1898523436-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d5efa20 at consistency null
2022-03-22 05:52:19.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1898523436-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:19.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1755791715-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47b13d09 at consistency null
2022-03-22 05:52:19.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1755791715-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:19.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396299733-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@222b522f at consistency null
2022-03-22 05:52:19.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396299733-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:19.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550709311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dbddf02 at consistency null
2022-03-22 05:52:19.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550709311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:20.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [412817218-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2728ad6f at consistency null
2022-03-22 05:52:20.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [412817218-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:20.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1248190827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@693f85f6 at consistency null
2022-03-22 05:52:20.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1248190827-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:20.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [628866135-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11fd18d4 at consistency null
2022-03-22 05:52:20.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [628866135-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:21.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475064855-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@426b9026 at consistency null
2022-03-22 05:52:21.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475064855-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:21.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555156952-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@204a1ed1 at consistency null
2022-03-22 05:52:21.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555156952-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:22.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [968491932-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35cf2f1b at consistency null
2022-03-22 05:52:22.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [968491932-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:22.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096518136-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c0daace at consistency null
2022-03-22 05:52:22.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096518136-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:22.922 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:52:28.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1522582422-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5279f9c9 at consistency null
2022-03-22 05:52:28.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1522582422-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:28.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [140314649-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2235d9c1 at consistency null
2022-03-22 05:52:28.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [140314649-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:28.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1636977773-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ece8d6b at consistency null
2022-03-22 05:52:28.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1636977773-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:28.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2090540526-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f3411a6 at consistency null
2022-03-22 05:52:28.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2090540526-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:28.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [538688920-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@521e1e38 at consistency null
2022-03-22 05:52:28.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [538688920-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:29.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1088840770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ee92e6b at consistency null
2022-03-22 05:52:29.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1088840770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:29.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [870623511-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44d64508 at consistency null
2022-03-22 05:52:29.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [870623511-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:30.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389330664-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b80d515 at consistency null
2022-03-22 05:52:30.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389330664-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:31.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1782460139-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@94df183 at consistency null
2022-03-22 05:52:31.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1782460139-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:31.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [420058010-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e400cb7 at consistency null
2022-03-22 05:52:31.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [420058010-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:31.922 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:52:36.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [263404989-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6848f472 at consistency null
2022-03-22 05:52:36.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [263404989-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:36.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1023564117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45cdbc13 at consistency null
2022-03-22 05:52:36.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1023564117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:37.090 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [651194728-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23ea9cc0 at consistency null
2022-03-22 05:52:37.090 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [651194728-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:37.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1938858859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dd119be at consistency null
2022-03-22 05:52:37.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1938858859-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:37.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1708668529-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12958112 at consistency null
2022-03-22 05:52:37.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1708668529-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:37.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1254036334-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a908a77 at consistency null
2022-03-22 05:52:37.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1254036334-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:38.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1866559584-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d6b1d79 at consistency null
2022-03-22 05:52:38.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1866559584-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:39.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431758616-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7842f183 at consistency null
2022-03-22 05:52:39.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431758616-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:39.951 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1039391712-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46616288 at consistency null
2022-03-22 05:52:39.951 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1039391712-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:40.768 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258313043-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@781c9b01 at consistency null
2022-03-22 05:52:40.768 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258313043-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:40.770 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:52:41.053 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:52:41.059 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:52:45.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1799567535-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d09d1ae at consistency null
2022-03-22 05:52:45.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1799567535-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:45.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [309242882-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@228fb65 at consistency null
2022-03-22 05:52:45.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [309242882-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:45.872 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [828459380-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f10bc8a at consistency null
2022-03-22 05:52:45.872 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [828459380-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:45.976 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310211352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53c8f2fa at consistency null
2022-03-22 05:52:45.976 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310211352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:46.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611245496-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@706fcd8c at consistency null
2022-03-22 05:52:46.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611245496-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:46.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [132126983-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3559f337 at consistency null
2022-03-22 05:52:46.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [132126983-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:47.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1014090342-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fe7edfa at consistency null
2022-03-22 05:52:47.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1014090342-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:48.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1173199683-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b2c9fde at consistency null
2022-03-22 05:52:48.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1173199683-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:48.314 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:52:53.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1760870433-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f9bab52 at consistency null
2022-03-22 05:52:53.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1760870433-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:53.438 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1868284953-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f530d66 at consistency null
2022-03-22 05:52:53.438 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1868284953-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:53.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1573585262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@500eab3d at consistency null
2022-03-22 05:52:53.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1573585262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:53.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1960164887-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ca7985a at consistency null
2022-03-22 05:52:53.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1960164887-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:53.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [988712292-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e2465a7 at consistency null
2022-03-22 05:52:53.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [988712292-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:54.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [195357996-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c297c6d at consistency null
2022-03-22 05:52:54.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [195357996-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:54.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [241257799-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c4c52fa at consistency null
2022-03-22 05:52:54.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [241257799-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:54.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469939610-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41180589 at consistency null
2022-03-22 05:52:54.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469939610-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:55.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [433479072-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23d8803d at consistency null
2022-03-22 05:52:55.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [433479072-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:56.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [901189179-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74454a3f at consistency null
2022-03-22 05:52:56.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [901189179-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:52:56.209 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:53:01.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825783999-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ae00dcf at consistency null
2022-03-22 05:53:01.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825783999-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:01.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [233156269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b556557 at consistency null
2022-03-22 05:53:01.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [233156269-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:01.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [653642220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cebd568 at consistency null
2022-03-22 05:53:01.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [653642220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:01.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [733928927-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32217e78 at consistency null
2022-03-22 05:53:01.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [733928927-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:01.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1497484185-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38259386 at consistency null
2022-03-22 05:53:01.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1497484185-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:02.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045871834-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18fa3608 at consistency null
2022-03-22 05:53:02.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045871834-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:02.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1908640417-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13e3897d at consistency null
2022-03-22 05:53:02.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1908640417-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:02.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [751759715-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74e9124 at consistency null
2022-03-22 05:53:02.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [751759715-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:03.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063919712-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e4dadb3 at consistency null
2022-03-22 05:53:03.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063919712-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:04.349 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [246834316-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23af0b30 at consistency null
2022-03-22 05:53:04.349 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [246834316-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:05.166 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1666015609-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c08d659 at consistency null
2022-03-22 05:53:05.166 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1666015609-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:05.180 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:53:10.186 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [945017242-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fbcb54d at consistency null
2022-03-22 05:53:10.186 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [945017242-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:10.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1691908890-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cbfbb3c at consistency null
2022-03-22 05:53:10.247 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1691908890-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:10.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1501028617-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2792d910 at consistency null
2022-03-22 05:53:10.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1501028617-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:10.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1371080183-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dc279a5 at consistency null
2022-03-22 05:53:10.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1371080183-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:10.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2000732507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77eeab8d at consistency null
2022-03-22 05:53:10.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2000732507-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:10.644 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1357345794-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76898817 at consistency null
2022-03-22 05:53:10.644 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1357345794-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:10.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914270398-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4762958f at consistency null
2022-03-22 05:53:10.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914270398-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:10.959 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804361531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19b3c397 at consistency null
2022-03-22 05:53:10.959 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804361531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:11.093 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:53:11.098 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:53:11.389 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1581962119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b83a1ef at consistency null
2022-03-22 05:53:11.389 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1581962119-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:11.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [895941416-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21f05bb8 at consistency null
2022-03-22 05:53:11.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [895941416-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:12.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981718114-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@726d6ce4 at consistency null
2022-03-22 05:53:12.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981718114-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:13.465 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1432716759-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75c5a645 at consistency null
2022-03-22 05:53:13.465 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1432716759-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:13.467 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:53:18.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572581911-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@734fedcb at consistency null
2022-03-22 05:53:18.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572581911-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:18.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1594277171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50414581 at consistency null
2022-03-22 05:53:18.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1594277171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:18.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1567216959-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49d21921 at consistency null
2022-03-22 05:53:18.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1567216959-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:18.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609990776-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11fd69d0 at consistency null
2022-03-22 05:53:18.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609990776-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:18.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619744750-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d50ef91 at consistency null
2022-03-22 05:53:18.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619744750-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:19.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1316307814-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47e7ccf7 at consistency null
2022-03-22 05:53:19.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1316307814-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:19.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1562814394-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e25de3b at consistency null
2022-03-22 05:53:19.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1562814394-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:19.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617045022-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64ec87c6 at consistency null
2022-03-22 05:53:19.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617045022-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:19.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83372693-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7232be5 at consistency null
2022-03-22 05:53:19.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83372693-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:19.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1618097951-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37c7f93c at consistency null
2022-03-22 05:53:19.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1618097951-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:19.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142690268-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37159ab5 at consistency null
2022-03-22 05:53:19.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142690268-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:19.955 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372821374-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f626b52 at consistency null
2022-03-22 05:53:19.955 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372821374-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:20.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1397879214-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1001758 at consistency null
2022-03-22 05:53:20.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1397879214-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:20.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [585825812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@906005d at consistency null
2022-03-22 05:53:20.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [585825812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:20.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542630450-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@125df3fa at consistency null
2022-03-22 05:53:20.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542630450-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:21.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1884588155-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e692218 at consistency null
2022-03-22 05:53:21.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1884588155-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:22.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743758686-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fd25c5a at consistency null
2022-03-22 05:53:22.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743758686-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:22.219 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:53:27.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1497074334-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26efa315 at consistency null
2022-03-22 05:53:27.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1497074334-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:27.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1189258745-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b505611 at consistency null
2022-03-22 05:53:27.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1189258745-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:27.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [304212494-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cfed681 at consistency null
2022-03-22 05:53:27.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [304212494-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:27.526 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [413490495-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c36bc8 at consistency null
2022-03-22 05:53:27.526 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [413490495-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:27.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [500128719-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5aef97ee at consistency null
2022-03-22 05:53:27.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [500128719-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:27.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1313990882-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@356b27c at consistency null
2022-03-22 05:53:27.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1313990882-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:28.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1404902076-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79e63649 at consistency null
2022-03-22 05:53:28.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1404902076-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:28.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1631844689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63618f96 at consistency null
2022-03-22 05:53:28.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1631844689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:28.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [112580501-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@459734f3 at consistency null
2022-03-22 05:53:28.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [112580501-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:28.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1206008153-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7364bb6f at consistency null
2022-03-22 05:53:28.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1206008153-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:29.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2083628003-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b788f34 at consistency null
2022-03-22 05:53:29.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2083628003-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:29.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [38159429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@540abaf2 at consistency null
2022-03-22 05:53:29.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [38159429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:29.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [719794919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28463c2c at consistency null
2022-03-22 05:53:29.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [719794919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:30.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2090466835-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e2f5588 at consistency null
2022-03-22 05:53:30.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2090466835-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:31.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [231109848-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1517456 at consistency null
2022-03-22 05:53:31.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [231109848-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:31.209 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:53:36.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1033006409-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d83a4 at consistency null
2022-03-22 05:53:36.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1033006409-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:36.294 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779157526-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@193368d5 at consistency null
2022-03-22 05:53:36.294 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779157526-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:36.375 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [489360045-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1db4fd6e at consistency null
2022-03-22 05:53:36.375 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [489360045-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:36.491 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603315556-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17cdc94a at consistency null
2022-03-22 05:53:36.491 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [603315556-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:36.597 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220402806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b3168b4 at consistency null
2022-03-22 05:53:36.597 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220402806-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:36.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [737461364-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79c6d2c7 at consistency null
2022-03-22 05:53:36.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [737461364-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:36.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1859116485-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43231f1c at consistency null
2022-03-22 05:53:36.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1859116485-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:36.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1675375493-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a5d09ab at consistency null
2022-03-22 05:53:36.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1675375493-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:37.050 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1992891180-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ad9a29c at consistency null
2022-03-22 05:53:37.051 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1992891180-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:37.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2073278181-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54611ea at consistency null
2022-03-22 05:53:37.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2073278181-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:37.383 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1805807015-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ec625a1 at consistency null
2022-03-22 05:53:37.384 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1805807015-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:37.587 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1767914317-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c847f83 at consistency null
2022-03-22 05:53:37.588 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1767914317-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:37.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1473137974-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bcee1cd at consistency null
2022-03-22 05:53:37.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1473137974-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:38.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [822934964-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3eb2681f at consistency null
2022-03-22 05:53:38.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [822934964-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:39.054 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1237021031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51b6b945 at consistency null
2022-03-22 05:53:39.054 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1237021031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:39.867 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568951118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e61ac97 at consistency null
2022-03-22 05:53:39.867 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568951118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:39.869 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:53:41.121 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:53:41.125 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:53:44.876 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74178488-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ddfa1f at consistency null
2022-03-22 05:53:44.876 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74178488-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:44.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1381786588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f0288cb at consistency null
2022-03-22 05:53:44.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1381786588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:44.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081999545-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b9a70ff at consistency null
2022-03-22 05:53:44.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081999545-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:44.964 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [770705929-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@386c680f at consistency null
2022-03-22 05:53:44.964 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [770705929-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:45.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2000459280-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@550c701f at consistency null
2022-03-22 05:53:45.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2000459280-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:45.099 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1179601320-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ef3fea5 at consistency null
2022-03-22 05:53:45.099 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1179601320-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:45.153 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1980031504-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7676660c at consistency null
2022-03-22 05:53:45.153 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1980031504-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:45.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535712923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d8144dd at consistency null
2022-03-22 05:53:45.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535712923-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:45.462 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1209442284-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bbb45fa at consistency null
2022-03-22 05:53:45.462 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1209442284-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:45.666 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957203279-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f4fcbbd at consistency null
2022-03-22 05:53:45.666 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957203279-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:45.882 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [627963304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8cfc525 at consistency null
2022-03-22 05:53:45.882 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [627963304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:46.087 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914578008-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@491ac553 at consistency null
2022-03-22 05:53:46.087 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914578008-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:46.298 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1242974875-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@476fc1f8 at consistency null
2022-03-22 05:53:46.298 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1242974875-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:46.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2061792355-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@162cf88b at consistency null
2022-03-22 05:53:46.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2061792355-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:47.135 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [820907554-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60bcf186 at consistency null
2022-03-22 05:53:47.136 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [820907554-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:47.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479450109-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f13909f at consistency null
2022-03-22 05:53:47.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479450109-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:48.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680412774-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@464aee84 at consistency null
2022-03-22 05:53:48.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680412774-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:48.419 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:53:53.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1027028972-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37d3be66 at consistency null
2022-03-22 05:53:53.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1027028972-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:53.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2104013174-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@306459db at consistency null
2022-03-22 05:53:53.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2104013174-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:53.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2137430301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5390a020 at consistency null
2022-03-22 05:53:53.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2137430301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:53.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407192247-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@423f5a18 at consistency null
2022-03-22 05:53:53.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407192247-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:54.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475135856-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6835ecb7 at consistency null
2022-03-22 05:53:54.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475135856-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:54.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1874272762-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a8ddaeb at consistency null
2022-03-22 05:53:54.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1874272762-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:54.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617800544-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ccaf43a at consistency null
2022-03-22 05:53:54.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617800544-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:55.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2005194489-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@141d8421 at consistency null
2022-03-22 05:53:55.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2005194489-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:56.088 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175594544-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@530a8241 at consistency null
2022-03-22 05:53:56.089 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175594544-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:56.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284981241-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47c45049 at consistency null
2022-03-22 05:53:56.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284981241-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:53:56.910 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:54:01.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596004709-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cd407af at consistency null
2022-03-22 05:54:01.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596004709-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:02.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919182578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4410a098 at consistency null
2022-03-22 05:54:02.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919182578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:02.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621870376-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@430f3ad1 at consistency null
2022-03-22 05:54:02.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621870376-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:02.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1444943308-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26ac6ce1 at consistency null
2022-03-22 05:54:02.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1444943308-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:02.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1204589834-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a59eee4 at consistency null
2022-03-22 05:54:02.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1204589834-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:02.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [374221047-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b8c8d57 at consistency null
2022-03-22 05:54:02.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [374221047-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:03.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [882208427-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@416fe945 at consistency null
2022-03-22 05:54:03.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [882208427-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:03.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1657639808-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4614eed at consistency null
2022-03-22 05:54:03.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1657639808-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:04.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091186198-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@713abe4b at consistency null
2022-03-22 05:54:04.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091186198-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:05.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [644136784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ec3730e at consistency null
2022-03-22 05:54:05.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [644136784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:05.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [958098689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50d88a52 at consistency null
2022-03-22 05:54:05.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [958098689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:05.842 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:54:10.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [935639089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e40a082 at consistency null
2022-03-22 05:54:10.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [935639089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:10.878 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [665737793-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7aac85e7 at consistency null
2022-03-22 05:54:10.879 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [665737793-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:10.940 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607387611-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f02d2bf at consistency null
2022-03-22 05:54:10.940 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607387611-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:11.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176694073-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b9c5317 at consistency null
2022-03-22 05:54:11.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176694073-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:11.076 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1412813052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4083b456 at consistency null
2022-03-22 05:54:11.076 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1412813052-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:11.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18674793-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e1e921c at consistency null
2022-03-22 05:54:11.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18674793-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:11.149 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:54:11.167 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:54:11.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [980033206-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4710344a at consistency null
2022-03-22 05:54:11.253 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [980033206-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:11.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1802528743-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e6dd354 at consistency null
2022-03-22 05:54:11.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1802528743-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:11.873 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [288946245-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6391f1cb at consistency null
2022-03-22 05:54:11.873 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [288946245-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:12.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2002909623-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f41dedf at consistency null
2022-03-22 05:54:12.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2002909623-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:14.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487904835-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fe65b10 at consistency null
2022-03-22 05:54:14.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487904835-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:14.327 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:54:19.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1966327129-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3395e68e at consistency null
2022-03-22 05:54:19.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1966327129-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:19.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589291882-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@524ff282 at consistency null
2022-03-22 05:54:19.461 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589291882-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:19.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531648255-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53f96932 at consistency null
2022-03-22 05:54:19.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531648255-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:19.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1616280347-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3eaa8686 at consistency null
2022-03-22 05:54:19.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1616280347-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:19.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [195326789-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2297b64b at consistency null
2022-03-22 05:54:19.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [195326789-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:20.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475139562-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f619b79 at consistency null
2022-03-22 05:54:20.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475139562-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:20.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415819528-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1749c35e at consistency null
2022-03-22 05:54:20.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415819528-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:20.531 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129930812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dbde987 at consistency null
2022-03-22 05:54:20.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129930812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:20.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [799601621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62872d95 at consistency null
2022-03-22 05:54:20.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [799601621-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:21.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [516563533-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61bec8ef at consistency null
2022-03-22 05:54:21.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [516563533-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:22.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1466448749-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38b4b4b5 at consistency null
2022-03-22 05:54:22.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1466448749-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:22.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613697738-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ad783f5 at consistency null
2022-03-22 05:54:22.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613697738-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:22.932 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:54:28.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1288319446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7167e8bf at consistency null
2022-03-22 05:54:28.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1288319446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:28.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [56474184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6689afcc at consistency null
2022-03-22 05:54:28.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [56474184-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:28.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [368593437-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b85aeb4 at consistency null
2022-03-22 05:54:28.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [368593437-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:28.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [321688176-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ddb421a at consistency null
2022-03-22 05:54:28.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [321688176-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:28.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250955074-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23b76d0e at consistency null
2022-03-22 05:54:28.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250955074-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:28.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611936046-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@86b417b at consistency null
2022-03-22 05:54:28.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611936046-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:28.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938162067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ecdaecc at consistency null
2022-03-22 05:54:28.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938162067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:28.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1084464140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6931ec9d at consistency null
2022-03-22 05:54:28.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1084464140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:29.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1691880975-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71f633a8 at consistency null
2022-03-22 05:54:29.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1691880975-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:29.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629464827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2344d474 at consistency null
2022-03-22 05:54:29.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629464827-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:29.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1270690423-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@104ca51e at consistency null
2022-03-22 05:54:29.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1270690423-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:30.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388142320-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@767fe68e at consistency null
2022-03-22 05:54:30.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388142320-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:31.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [858671213-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3654e09a at consistency null
2022-03-22 05:54:31.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [858671213-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:31.139 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:54:36.162 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [819866613-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33b66550 at consistency null
2022-03-22 05:54:36.162 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [819866613-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:36.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315787539-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4627426d at consistency null
2022-03-22 05:54:36.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315787539-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:36.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [290724525-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f6597b1 at consistency null
2022-03-22 05:54:36.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [290724525-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:36.447 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1358452170-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e24fef7 at consistency null
2022-03-22 05:54:36.447 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1358452170-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:36.651 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [297291171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5eb88c86 at consistency null
2022-03-22 05:54:36.652 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [297291171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:36.859 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396674198-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22f09916 at consistency null
2022-03-22 05:54:36.859 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [396674198-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:37.268 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [647243197-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@414b5ae6 at consistency null
2022-03-22 05:54:37.268 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [647243197-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:38.082 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [886076064-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38e1c77e at consistency null
2022-03-22 05:54:38.082 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [886076064-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:39.697 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742140620-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22a7a058 at consistency null
2022-03-22 05:54:39.698 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742140620-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:39.700 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:54:41.179 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:54:41.186 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:54:44.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1457986772-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@439a533 at consistency null
2022-03-22 05:54:44.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1457986772-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:44.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373201016-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fbe5338 at consistency null
2022-03-22 05:54:44.743 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373201016-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:44.786 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1714114449-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2952bd86 at consistency null
2022-03-22 05:54:44.786 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1714114449-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:44.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906733324-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36687d94 at consistency null
2022-03-22 05:54:44.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906733324-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:44.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1482955941-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fb14657 at consistency null
2022-03-22 05:54:44.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1482955941-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:44.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1119125363-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@127bfd0a at consistency null
2022-03-22 05:54:44.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1119125363-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:45.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128602995-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e79dc4e at consistency null
2022-03-22 05:54:45.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128602995-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:45.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361176163-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b0880c8 at consistency null
2022-03-22 05:54:45.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361176163-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:45.647 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1837017922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e7ee600 at consistency null
2022-03-22 05:54:45.647 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1837017922-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:46.056 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214961785-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63a0dd68 at consistency null
2022-03-22 05:54:46.056 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214961785-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:46.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229999855-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dbe6fe at consistency null
2022-03-22 05:54:46.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229999855-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:48.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [553444670-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c4a388d at consistency null
2022-03-22 05:54:48.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [553444670-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:48.538 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:54:53.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [943750872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45dd93e2 at consistency null
2022-03-22 05:54:53.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [943750872-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:53.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [888081519-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20f35dac at consistency null
2022-03-22 05:54:53.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [888081519-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:53.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366769377-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c43b35a at consistency null
2022-03-22 05:54:53.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366769377-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:53.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1389278833-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3180720f at consistency null
2022-03-22 05:54:53.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1389278833-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:54.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1774429262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18eba55e at consistency null
2022-03-22 05:54:54.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1774429262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:54.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629363053-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22cf95d2 at consistency null
2022-03-22 05:54:54.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629363053-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:54.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1393443392-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43f73bb2 at consistency null
2022-03-22 05:54:54.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1393443392-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:54.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1960053614-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@715529c9 at consistency null
2022-03-22 05:54:54.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1960053614-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:55.631 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [4268142-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3451a3e8 at consistency null
2022-03-22 05:54:55.631 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [4268142-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:57.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1073086770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6aeec1b5 at consistency null
2022-03-22 05:54:57.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1073086770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:54:57.316 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:55:02.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [516643022-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@103e5df1 at consistency null
2022-03-22 05:55:02.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [516643022-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:02.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1242607362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@487d255e at consistency null
2022-03-22 05:55:02.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1242607362-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:02.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138472062-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@718a815a at consistency null
2022-03-22 05:55:02.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138472062-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:02.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286324902-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6df515b7 at consistency null
2022-03-22 05:55:02.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286324902-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:03.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [149090005-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fc61c2d at consistency null
2022-03-22 05:55:03.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [149090005-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:03.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1195718772-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e308f5e at consistency null
2022-03-22 05:55:03.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1195718772-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:03.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495473318-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b155b93 at consistency null
2022-03-22 05:55:03.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495473318-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:04.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267682262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e88bfeb at consistency null
2022-03-22 05:55:04.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267682262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:04.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1409581108-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7110e563 at consistency null
2022-03-22 05:55:04.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1409581108-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:05.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [980190296-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a18cf4e at consistency null
2022-03-22 05:55:05.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [980190296-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:06.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2070636959-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71645001 at consistency null
2022-03-22 05:55:06.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2070636959-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:06.223 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:55:06.746 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:55:06.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-45, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 05:55:06.752 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-45, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 05:55:06.752 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-45, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:55:06.754 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-45, inFlight=0, closed=true] closing connection
2022-03-22 05:55:06.754 DEBUG 10 --- [uster-worker-81] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 05:55:06.756 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 05:55:06.756 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 600000 ms
2022-03-22 05:55:11.210 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:55:11.214 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:55:11.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1242258655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@351ea3cb at consistency null
2022-03-22 05:55:11.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1242258655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:11.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [989753482-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7eb949db at consistency null
2022-03-22 05:55:11.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [989753482-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:11.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693347273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a058be1 at consistency null
2022-03-22 05:55:11.394 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693347273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:11.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1147783002-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@341890cd at consistency null
2022-03-22 05:55:11.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1147783002-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:11.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1419019948-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74d86d7e at consistency null
2022-03-22 05:55:11.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1419019948-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:11.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497538708-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@409b15af at consistency null
2022-03-22 05:55:11.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497538708-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:12.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179335372-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76f7c4de at consistency null
2022-03-22 05:55:12.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179335372-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:12.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15279575-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@699ac35 at consistency null
2022-03-22 05:55:12.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15279575-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:12.739 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1891136128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@311d9205 at consistency null
2022-03-22 05:55:12.739 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1891136128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:13.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401428341-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29b3dfe4 at consistency null
2022-03-22 05:55:13.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401428341-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:13.571 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1521956312-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61aa3f38 at consistency null
2022-03-22 05:55:13.571 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1521956312-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:13.994 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [330917060-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a19df3c at consistency null
2022-03-22 05:55:13.995 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [330917060-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:14.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [539371585-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77c9fdc0 at consistency null
2022-03-22 05:55:14.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [539371585-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:14.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [863428225-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40325022 at consistency null
2022-03-22 05:55:14.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [863428225-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:14.838 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:55:19.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1351365826-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70d1f4e3 at consistency null
2022-03-22 05:55:19.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1351365826-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:20.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139195731-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@145535fa at consistency null
2022-03-22 05:55:20.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139195731-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:20.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [423534651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ccafd4d at consistency null
2022-03-22 05:55:20.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [423534651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:20.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [473556811-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7eab0bb6 at consistency null
2022-03-22 05:55:20.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [473556811-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:20.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [765651498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a28e6c at consistency null
2022-03-22 05:55:20.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [765651498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:20.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2056549280-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c658147 at consistency null
2022-03-22 05:55:20.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2056549280-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:20.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [382155697-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@473e2313 at consistency null
2022-03-22 05:55:20.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [382155697-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:20.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1159169437-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@616e76f9 at consistency null
2022-03-22 05:55:20.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1159169437-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:20.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136921263-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@742a4556 at consistency null
2022-03-22 05:55:20.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136921263-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:20.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [646881687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64d58d2d at consistency null
2022-03-22 05:55:20.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [646881687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:21.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [163257412-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52c51c1f at consistency null
2022-03-22 05:55:21.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [163257412-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:21.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1895390083-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@400e7100 at consistency null
2022-03-22 05:55:21.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1895390083-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:21.834 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2039157407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66a88ca2 at consistency null
2022-03-22 05:55:21.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2039157407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:22.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718964256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2edcc9bf at consistency null
2022-03-22 05:55:22.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718964256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:22.755 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2077423107-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54830d9f at consistency null
2022-03-22 05:55:22.755 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2077423107-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:23.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1103990713-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6736f07e at consistency null
2022-03-22 05:55:23.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1103990713-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:23.644 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:55:28.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626563317-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d001583 at consistency null
2022-03-22 05:55:28.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626563317-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:28.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510006093-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41a30e49 at consistency null
2022-03-22 05:55:28.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510006093-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:28.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919454316-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11c6b303 at consistency null
2022-03-22 05:55:28.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919454316-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:29.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1601529311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19fa0800 at consistency null
2022-03-22 05:55:29.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1601529311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:29.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619331598-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@145bbab1 at consistency null
2022-03-22 05:55:29.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619331598-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:29.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179880869-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dd33f35 at consistency null
2022-03-22 05:55:29.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179880869-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:29.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1006974322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6929d327 at consistency null
2022-03-22 05:55:29.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1006974322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:29.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1219163312-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20da6c0a at consistency null
2022-03-22 05:55:29.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1219163312-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:29.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1043740571-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@691dd492 at consistency null
2022-03-22 05:55:29.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1043740571-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:29.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1927245331-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d636413 at consistency null
2022-03-22 05:55:29.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1927245331-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:29.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1734638495-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1af43873 at consistency null
2022-03-22 05:55:29.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1734638495-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:30.149 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029264055-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69bdbd1b at consistency null
2022-03-22 05:55:30.149 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029264055-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:30.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2003924701-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e835dfa at consistency null
2022-03-22 05:55:30.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2003924701-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:31.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128028118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48e6efc5 at consistency null
2022-03-22 05:55:31.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128028118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:31.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [984290576-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@736a710a at consistency null
2022-03-22 05:55:31.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [984290576-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:32.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1650636630-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@773b5f19 at consistency null
2022-03-22 05:55:32.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1650636630-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:32.707 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:55:37.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [509207129-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@323a089 at consistency null
2022-03-22 05:55:37.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [509207129-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:37.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1497872853-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cc4efef at consistency null
2022-03-22 05:55:37.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1497872853-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:37.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1000744056-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c86335c at consistency null
2022-03-22 05:55:37.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1000744056-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:37.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1275718749-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71686c42 at consistency null
2022-03-22 05:55:37.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1275718749-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:38.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487822511-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fd1ad0e at consistency null
2022-03-22 05:55:38.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487822511-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:38.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046697417-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34b19c8c at consistency null
2022-03-22 05:55:38.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046697417-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:38.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1945198954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59c05595 at consistency null
2022-03-22 05:55:38.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1945198954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:38.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1880686669-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bf82893 at consistency null
2022-03-22 05:55:38.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1880686669-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:39.045 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1907246093-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@923413f at consistency null
2022-03-22 05:55:39.045 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1907246093-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:39.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2024213926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31cee96b at consistency null
2022-03-22 05:55:39.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2024213926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:40.522 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 05:55:40.528 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-69, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 05:55:40.528 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-69, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 05:55:40.528 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-69, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 05:55:40.529 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-69, inFlight=0, closed=true] closing connection
2022-03-22 05:55:40.529 DEBUG 10 --- [uster-worker-82] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 05:55:40.529 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 05:55:40.529 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 600000 ms
2022-03-22 05:55:40.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [10869225-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6af96e3a at consistency null
2022-03-22 05:55:40.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [10869225-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:41.242 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:55:41.246 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:55:41.549 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [948349656-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40ef7f46 at consistency null
2022-03-22 05:55:41.549 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [948349656-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:41.550 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:55:46.631 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998969321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53d6e4dd at consistency null
2022-03-22 05:55:46.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998969321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:46.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2072112899-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1595da7d at consistency null
2022-03-22 05:55:46.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2072112899-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:46.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1438980875-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@343cfe4b at consistency null
2022-03-22 05:55:46.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1438980875-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:46.957 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1478011649-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f12b2c5 at consistency null
2022-03-22 05:55:46.957 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1478011649-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:47.067 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [990843820-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b065ff6 at consistency null
2022-03-22 05:55:47.067 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [990843820-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:47.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1333922361-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@507b26d9 at consistency null
2022-03-22 05:55:47.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1333922361-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:47.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [853691777-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e6443b at consistency null
2022-03-22 05:55:47.341 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [853691777-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:47.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [792705151-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19785a18 at consistency null
2022-03-22 05:55:47.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [792705151-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:48.053 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1819171911-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40d84f2e at consistency null
2022-03-22 05:55:48.053 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1819171911-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:48.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862081073-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@491ce402 at consistency null
2022-03-22 05:55:48.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862081073-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:49.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [323788185-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78eaa686 at consistency null
2022-03-22 05:55:49.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [323788185-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:49.411 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:55:54.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096954831-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23998b19 at consistency null
2022-03-22 05:55:54.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096954831-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:54.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1601229760-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21ba117b at consistency null
2022-03-22 05:55:54.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1601229760-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:54.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2125657031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fa8cdba at consistency null
2022-03-22 05:55:54.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2125657031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:54.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [166009309-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22db945f at consistency null
2022-03-22 05:55:54.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [166009309-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:54.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1498001253-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4df5125e at consistency null
2022-03-22 05:55:54.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1498001253-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:55.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [16685407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cb5f352 at consistency null
2022-03-22 05:55:55.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [16685407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:56.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1773895244-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75b93c63 at consistency null
2022-03-22 05:55:56.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1773895244-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:56.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1219602775-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b3e27e7 at consistency null
2022-03-22 05:55:56.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1219602775-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:56.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553405347-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e3cb33c at consistency null
2022-03-22 05:55:56.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553405347-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:56.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [676809487-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d00e2ca at consistency null
2022-03-22 05:55:56.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [676809487-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:57.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [447482199-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41e6832d at consistency null
2022-03-22 05:55:57.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [447482199-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:57.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603461188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@303364f5 at consistency null
2022-03-22 05:55:57.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603461188-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:55:57.910 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:56:02.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [994888728-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4adfc68b at consistency null
2022-03-22 05:56:02.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [994888728-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:03.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [693237581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f55cb16 at consistency null
2022-03-22 05:56:03.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [693237581-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:03.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1405986177-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68b033c8 at consistency null
2022-03-22 05:56:03.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1405986177-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:03.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [454426492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@430fe6e7 at consistency null
2022-03-22 05:56:03.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [454426492-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:03.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181015050-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77d08b93 at consistency null
2022-03-22 05:56:03.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181015050-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:03.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1613340692-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66cce304 at consistency null
2022-03-22 05:56:03.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1613340692-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:04.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1593241661-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f0ba266 at consistency null
2022-03-22 05:56:04.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1593241661-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:04.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1789663872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@338f45be at consistency null
2022-03-22 05:56:04.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1789663872-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:05.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [172498771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4070cc4f at consistency null
2022-03-22 05:56:05.044 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [172498771-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:05.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [667694862-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4125c386 at consistency null
2022-03-22 05:56:05.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [667694862-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:05.911 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:56:10.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1322405074-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@136f4156 at consistency null
2022-03-22 05:56:10.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1322405074-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:11.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151350674-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a045430 at consistency null
2022-03-22 05:56:11.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151350674-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:11.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [240655058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c50e98b at consistency null
2022-03-22 05:56:11.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [240655058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:11.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223309392-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73063074 at consistency null
2022-03-22 05:56:11.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223309392-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:11.302 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:56:11.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [335698243-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a0abab3 at consistency null
2022-03-22 05:56:11.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [335698243-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:11.338 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:56:11.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373364470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6257d816 at consistency null
2022-03-22 05:56:11.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373364470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:11.494 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [521928341-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79507966 at consistency null
2022-03-22 05:56:11.494 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [521928341-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:11.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [103447966-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b1763a8 at consistency null
2022-03-22 05:56:11.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [103447966-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:11.851 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15000665-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7407895b at consistency null
2022-03-22 05:56:11.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15000665-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:12.263 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [238450915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@284c78a5 at consistency null
2022-03-22 05:56:12.264 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [238450915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:12.677 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318703334-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@522dbf83 at consistency null
2022-03-22 05:56:12.685 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318703334-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:13.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [462062118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52279c4d at consistency null
2022-03-22 05:56:13.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [462062118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:13.954 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1022061346-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a228a03 at consistency null
2022-03-22 05:56:13.954 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1022061346-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:13.956 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:56:19.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [19393852-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@516050b2 at consistency null
2022-03-22 05:56:19.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [19393852-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:19.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [20513566-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e8e3cba at consistency null
2022-03-22 05:56:19.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [20513566-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:19.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324169901-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2510d2 at consistency null
2022-03-22 05:56:19.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324169901-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:19.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [699984425-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e59e047 at consistency null
2022-03-22 05:56:19.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [699984425-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:19.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390638639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d6fd4fe at consistency null
2022-03-22 05:56:19.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390638639-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:19.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579183147-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@429b5805 at consistency null
2022-03-22 05:56:19.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579183147-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:19.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2042567952-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a906304 at consistency null
2022-03-22 05:56:19.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2042567952-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:20.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397957548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b8aedc9 at consistency null
2022-03-22 05:56:20.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397957548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:20.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1162065568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8610462 at consistency null
2022-03-22 05:56:20.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1162065568-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:21.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417319703-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10222e1b at consistency null
2022-03-22 05:56:21.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [417319703-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:21.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [420024961-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7943e758 at consistency null
2022-03-22 05:56:21.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [420024961-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:22.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1277779253-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dbb91c at consistency null
2022-03-22 05:56:22.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1277779253-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:22.430 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:56:27.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344396966-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@599ffd1 at consistency null
2022-03-22 05:56:27.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [344396966-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:27.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919345234-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ca4b8f at consistency null
2022-03-22 05:56:27.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919345234-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:27.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598371205-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62550569 at consistency null
2022-03-22 05:56:27.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598371205-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:27.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [627857968-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a1e4366 at consistency null
2022-03-22 05:56:27.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [627857968-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:28.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [157440031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70335483 at consistency null
2022-03-22 05:56:28.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [157440031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:28.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1280532243-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2673ff39 at consistency null
2022-03-22 05:56:28.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1280532243-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:28.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1020650335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cc7d8d3 at consistency null
2022-03-22 05:56:28.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1020650335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:28.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066711343-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@240aa35e at consistency null
2022-03-22 05:56:28.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066711343-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:28.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620479733-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e776c10 at consistency null
2022-03-22 05:56:28.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620479733-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:29.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921431603-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77c4d8d0 at consistency null
2022-03-22 05:56:29.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921431603-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:29.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632761514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@259797bb at consistency null
2022-03-22 05:56:29.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632761514-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:29.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579265992-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5476d004 at consistency null
2022-03-22 05:56:29.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579265992-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:30.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1991452235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21943176 at consistency null
2022-03-22 05:56:30.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1991452235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:30.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532731096-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70c76fcc at consistency null
2022-03-22 05:56:30.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532731096-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:30.814 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:56:35.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [59668982-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69740420 at consistency null
2022-03-22 05:56:35.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [59668982-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:36.035 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372466922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@be01ffa at consistency null
2022-03-22 05:56:36.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372466922-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:36.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2044561689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@408d27dd at consistency null
2022-03-22 05:56:36.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2044561689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:36.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1621288174-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9917ee8 at consistency null
2022-03-22 05:56:36.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1621288174-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:36.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269709509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@750e5bc6 at consistency null
2022-03-22 05:56:36.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269709509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:37.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487215641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c942975 at consistency null
2022-03-22 05:56:37.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487215641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:38.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [143295968-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e707a31 at consistency null
2022-03-22 05:56:38.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [143295968-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:39.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146997643-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71a5ccc7 at consistency null
2022-03-22 05:56:39.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146997643-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:39.933 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:56:41.340 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:56:41.407 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:56:44.942 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1961248745-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7672795e at consistency null
2022-03-22 05:56:44.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1961248745-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:45.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [993217014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b502964 at consistency null
2022-03-22 05:56:45.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [993217014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:45.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741319925-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56ac6bcd at consistency null
2022-03-22 05:56:45.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741319925-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:45.256 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1554381642-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@930462b at consistency null
2022-03-22 05:56:45.256 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1554381642-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:45.491 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [440018488-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c6d66da at consistency null
2022-03-22 05:56:45.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [440018488-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:45.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005035632-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@667c793a at consistency null
2022-03-22 05:56:45.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005035632-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:46.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258919487-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@533b843b at consistency null
2022-03-22 05:56:46.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258919487-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:46.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2086799667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58318169 at consistency null
2022-03-22 05:56:46.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2086799667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:47.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [933428065-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f6d7703 at consistency null
2022-03-22 05:56:47.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [933428065-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:48.650 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [834681285-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a5d707d at consistency null
2022-03-22 05:56:48.650 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [834681285-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:48.651 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:56:53.673 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211503862-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55483d51 at consistency null
2022-03-22 05:56:53.673 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211503862-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:53.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619816786-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@126c458 at consistency null
2022-03-22 05:56:53.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [619816786-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:53.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [78133598-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35477e10 at consistency null
2022-03-22 05:56:53.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [78133598-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:54.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873870879-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4282e5d at consistency null
2022-03-22 05:56:54.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873870879-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:54.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1283392975-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@109e393c at consistency null
2022-03-22 05:56:54.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1283392975-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:54.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1704845349-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27ad3b73 at consistency null
2022-03-22 05:56:54.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1704845349-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:54.474 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540694273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29947668 at consistency null
2022-03-22 05:56:54.474 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540694273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:54.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622410711-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c8bffc1 at consistency null
2022-03-22 05:56:54.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622410711-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:55.047 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [293340102-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d3d6cb5 at consistency null
2022-03-22 05:56:55.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [293340102-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:55.364 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [468337924-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4305b8c2 at consistency null
2022-03-22 05:56:55.364 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [468337924-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:55.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1771735847-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4928a1fc at consistency null
2022-03-22 05:56:55.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1771735847-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:56.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [87147072-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7aee2aef at consistency null
2022-03-22 05:56:56.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [87147072-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:56.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1948422931-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c50a9c3 at consistency null
2022-03-22 05:56:56.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1948422931-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:57.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026112246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72fdc123 at consistency null
2022-03-22 05:56:57.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026112246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:57.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5725639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ed6a9c1 at consistency null
2022-03-22 05:56:57.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5725639-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:56:57.522 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:57:02.526 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [201309263-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75e44da6 at consistency null
2022-03-22 05:57:02.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [201309263-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:02.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797594152-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c63fc68 at consistency null
2022-03-22 05:57:02.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797594152-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:02.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523203888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@700bdc1e at consistency null
2022-03-22 05:57:02.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523203888-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:02.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [623684215-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b58fb7f at consistency null
2022-03-22 05:57:02.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [623684215-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:03.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1397188441-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c0608cf at consistency null
2022-03-22 05:57:03.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1397188441-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:03.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [272688453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ff5299b at consistency null
2022-03-22 05:57:03.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [272688453-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:03.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1956811912-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31b0fd57 at consistency null
2022-03-22 05:57:03.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1956811912-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:04.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [874379129-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dcf7b1d at consistency null
2022-03-22 05:57:04.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [874379129-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:04.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552429836-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1153521b at consistency null
2022-03-22 05:57:04.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552429836-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:05.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1274388245-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27dc185b at consistency null
2022-03-22 05:57:05.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1274388245-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:05.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [731071543-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@113003b2 at consistency null
2022-03-22 05:57:05.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [731071543-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:06.472 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886312451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2791951a at consistency null
2022-03-22 05:57:06.472 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886312451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:06.481 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:57:11.408 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:57:11.440 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:57:11.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835242833-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cd9d879 at consistency null
2022-03-22 05:57:11.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835242833-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:11.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [176792287-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b001379 at consistency null
2022-03-22 05:57:11.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [176792287-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:11.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998964549-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a3ca038 at consistency null
2022-03-22 05:57:11.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998964549-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:11.846 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025676535-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51aed320 at consistency null
2022-03-22 05:57:11.846 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025676535-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:11.979 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1285179167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33d6c37c at consistency null
2022-03-22 05:57:11.980 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1285179167-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:12.190 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [197298953-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@582695a4 at consistency null
2022-03-22 05:57:12.190 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [197298953-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:12.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741565921-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60143ffb at consistency null
2022-03-22 05:57:12.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741565921-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:12.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315084995-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65dccda6 at consistency null
2022-03-22 05:57:12.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315084995-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:13.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348378287-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9bbef0c at consistency null
2022-03-22 05:57:13.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348378287-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:14.094 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1331416430-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68c1e6d0 at consistency null
2022-03-22 05:57:14.095 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1331416430-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:14.096 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:57:19.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1501315651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@296be7d4 at consistency null
2022-03-22 05:57:19.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1501315651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:19.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [51543853-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ad9aea2 at consistency null
2022-03-22 05:57:19.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [51543853-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:19.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [656220678-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ccce77a at consistency null
2022-03-22 05:57:19.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [656220678-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:19.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [909945654-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b5c6955 at consistency null
2022-03-22 05:57:19.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [909945654-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:20.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [233413096-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d7f68ad at consistency null
2022-03-22 05:57:20.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [233413096-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:20.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487748719-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a34671e at consistency null
2022-03-22 05:57:20.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [487748719-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:21.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406236889-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1158a615 at consistency null
2022-03-22 05:57:21.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406236889-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:21.739 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [6753422-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@693663aa at consistency null
2022-03-22 05:57:21.740 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [6753422-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:22.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098965939-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f1a4421 at consistency null
2022-03-22 05:57:22.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098965939-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:22.623 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:57:27.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1329551072-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56bfc04b at consistency null
2022-03-22 05:57:27.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1329551072-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:27.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [401668206-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@283ac87 at consistency null
2022-03-22 05:57:27.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [401668206-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:27.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [793482594-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58f2b44e at consistency null
2022-03-22 05:57:27.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [793482594-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:28.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1771075139-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75e4c83d at consistency null
2022-03-22 05:57:28.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1771075139-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:28.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1554884039-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42098bff at consistency null
2022-03-22 05:57:28.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1554884039-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:28.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1284375988-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f940734 at consistency null
2022-03-22 05:57:28.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1284375988-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:28.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1058896322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c6283db at consistency null
2022-03-22 05:57:28.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1058896322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:28.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1440031244-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@edb1076 at consistency null
2022-03-22 05:57:28.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1440031244-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:28.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692874476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33421 at consistency null
2022-03-22 05:57:28.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692874476-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:29.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1492659297-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f012eb2 at consistency null
2022-03-22 05:57:29.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1492659297-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:29.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [209690069-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67350cd2 at consistency null
2022-03-22 05:57:29.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [209690069-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:30.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1883642644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31bbf8e2 at consistency null
2022-03-22 05:57:30.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1883642644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:30.319 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:57:35.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768279462-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@288070c8 at consistency null
2022-03-22 05:57:35.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768279462-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:35.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [885635418-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ed2bfd4 at consistency null
2022-03-22 05:57:35.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [885635418-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:35.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151863587-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@414e7140 at consistency null
2022-03-22 05:57:35.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151863587-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:35.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83240473-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f5cfd63 at consistency null
2022-03-22 05:57:35.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83240473-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:36.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1901450002-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@471de70 at consistency null
2022-03-22 05:57:36.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1901450002-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:36.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [145729291-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fc6fd35 at consistency null
2022-03-22 05:57:36.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [145729291-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:36.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [207242286-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2db04bfc at consistency null
2022-03-22 05:57:36.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [207242286-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:37.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [967404882-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@654f0ccc at consistency null
2022-03-22 05:57:37.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [967404882-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:37.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1715201078-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fc7cc4b at consistency null
2022-03-22 05:57:37.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1715201078-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:38.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [333021189-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47c0abd3 at consistency null
2022-03-22 05:57:38.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [333021189-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:38.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1300024676-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6800f77c at consistency null
2022-03-22 05:57:38.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1300024676-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:38.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070459492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75b4afdc at consistency null
2022-03-22 05:57:38.893 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070459492-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:38.894 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:57:41.441 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:57:41.447 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:57:43.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [107918335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24c090b5 at consistency null
2022-03-22 05:57:43.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [107918335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:43.981 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1500037791-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@669c29cd at consistency null
2022-03-22 05:57:43.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1500037791-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:44.056 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1918116667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9c88975 at consistency null
2022-03-22 05:57:44.056 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1918116667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:44.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [572156383-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59d21ddf at consistency null
2022-03-22 05:57:44.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [572156383-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:44.189 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1368971048-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79376b68 at consistency null
2022-03-22 05:57:44.192 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1368971048-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:44.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1316723646-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e62e7c2 at consistency null
2022-03-22 05:57:44.267 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1316723646-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:44.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493171889-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@332c4a5a at consistency null
2022-03-22 05:57:44.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493171889-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:44.547 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1824580179-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ad98505 at consistency null
2022-03-22 05:57:44.547 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1824580179-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:44.655 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2071353716-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38f7cd1d at consistency null
2022-03-22 05:57:44.655 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2071353716-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:44.795 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1774357455-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51c26136 at consistency null
2022-03-22 05:57:44.795 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1774357455-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:45.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572268794-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d8fce at consistency null
2022-03-22 05:57:45.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572268794-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:45.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906971493-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@362ac36b at consistency null
2022-03-22 05:57:45.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906971493-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:45.459 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2147158825-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30337f96 at consistency null
2022-03-22 05:57:45.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2147158825-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:45.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43674602-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72ba7377 at consistency null
2022-03-22 05:57:45.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43674602-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:45.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754374265-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e458111 at consistency null
2022-03-22 05:57:45.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754374265-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:46.165 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [614041711-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e142829 at consistency null
2022-03-22 05:57:46.165 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [614041711-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:46.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60583318-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55f027b2 at consistency null
2022-03-22 05:57:46.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60583318-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:46.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1376289588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74f0c7b0 at consistency null
2022-03-22 05:57:46.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1376289588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:47.055 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [931095396-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@468f8d36 at consistency null
2022-03-22 05:57:47.056 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [931095396-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:47.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1598902036-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1905b0ae at consistency null
2022-03-22 05:57:47.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1598902036-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:47.509 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:57:52.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1071479222-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d1cd98b at consistency null
2022-03-22 05:57:52.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1071479222-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:52.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [693787738-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f235adf at consistency null
2022-03-22 05:57:52.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [693787738-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:52.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1710390870-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ff9630 at consistency null
2022-03-22 05:57:52.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1710390870-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:52.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349695507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31cbc6e1 at consistency null
2022-03-22 05:57:52.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349695507-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:53.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [70675298-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ab091d2 at consistency null
2022-03-22 05:57:53.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [70675298-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:53.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2138398514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fe0d5f9 at consistency null
2022-03-22 05:57:53.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2138398514-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:53.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [151227972-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46d78a23 at consistency null
2022-03-22 05:57:53.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [151227972-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:53.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282674007-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2944b0d6 at consistency null
2022-03-22 05:57:53.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282674007-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:54.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2075005632-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37648a2d at consistency null
2022-03-22 05:57:54.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2075005632-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:55.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779570760-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e12de25 at consistency null
2022-03-22 05:57:55.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779570760-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:55.967 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [999523436-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@746ecb56 at consistency null
2022-03-22 05:57:55.967 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [999523436-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:57:55.972 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:58:01.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1465985276-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78a0170d at consistency null
2022-03-22 05:58:01.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1465985276-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:01.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1557272770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b86dfd6 at consistency null
2022-03-22 05:58:01.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1557272770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:01.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1220745246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73fdd775 at consistency null
2022-03-22 05:58:01.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1220745246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:01.337 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1876182414-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@714b9627 at consistency null
2022-03-22 05:58:01.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1876182414-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:01.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825013824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9abef7c at consistency null
2022-03-22 05:58:01.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825013824-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:01.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2113186991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@540ef9f0 at consistency null
2022-03-22 05:58:01.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2113186991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:01.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [765941991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49d5da7b at consistency null
2022-03-22 05:58:01.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [765941991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:02.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611825384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@648287c8 at consistency null
2022-03-22 05:58:02.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611825384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:02.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [886617116-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3204442b at consistency null
2022-03-22 05:58:02.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [886617116-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:03.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1813112828-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1166eae2 at consistency null
2022-03-22 05:58:03.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1813112828-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:03.441 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:58:08.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2078735097-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7687ecb7 at consistency null
2022-03-22 05:58:08.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2078735097-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:08.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [260513965-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cb8968 at consistency null
2022-03-22 05:58:08.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [260513965-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:08.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123238677-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@167eb086 at consistency null
2022-03-22 05:58:08.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123238677-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:08.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [499329855-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30696951 at consistency null
2022-03-22 05:58:08.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [499329855-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:08.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1371417420-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7942c9bf at consistency null
2022-03-22 05:58:08.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1371417420-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:09.050 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728768926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23988f36 at consistency null
2022-03-22 05:58:09.050 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728768926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:09.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776461007-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4001f143 at consistency null
2022-03-22 05:58:09.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776461007-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:09.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836900383-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bd93bf3 at consistency null
2022-03-22 05:58:09.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836900383-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:09.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1295532727-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54b91206 at consistency null
2022-03-22 05:58:09.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1295532727-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:09.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103318165-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33bfd698 at consistency null
2022-03-22 05:58:09.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103318165-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:09.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [31854893-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c7bb245 at consistency null
2022-03-22 05:58:09.644 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [31854893-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:09.886 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [185623830-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e0f189a at consistency null
2022-03-22 05:58:09.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [185623830-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:10.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [70431545-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@123a127f at consistency null
2022-03-22 05:58:10.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [70431545-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:10.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165013396-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ffc0db6 at consistency null
2022-03-22 05:58:10.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165013396-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:10.541 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [940306682-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7eb46596 at consistency null
2022-03-22 05:58:10.541 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [940306682-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:10.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [817564216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8096912 at consistency null
2022-03-22 05:58:10.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [817564216-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:11.472 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:58:11.480 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:58:11.785 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [562158701-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a622b86 at consistency null
2022-03-22 05:58:11.785 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [562158701-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:11.794 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:58:16.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [130967182-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75cc1ec1 at consistency null
2022-03-22 05:58:16.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [130967182-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:16.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1621478061-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28894d3b at consistency null
2022-03-22 05:58:16.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1621478061-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:16.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [145225291-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b8bf0c1 at consistency null
2022-03-22 05:58:16.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [145225291-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:16.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [661266822-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@306ed82d at consistency null
2022-03-22 05:58:16.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [661266822-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:17.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407506931-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25c1e2f7 at consistency null
2022-03-22 05:58:17.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407506931-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:17.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [500863360-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@474393e1 at consistency null
2022-03-22 05:58:17.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [500863360-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:17.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720906718-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@737a6e30 at consistency null
2022-03-22 05:58:17.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720906718-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:17.267 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550342934-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35137fc2 at consistency null
2022-03-22 05:58:17.267 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550342934-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:17.378 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [699804201-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fefe5cd at consistency null
2022-03-22 05:58:17.378 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [699804201-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:17.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1613359118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4468b14a at consistency null
2022-03-22 05:58:17.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1613359118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:18.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [192191244-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64c34c19 at consistency null
2022-03-22 05:58:18.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [192191244-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:18.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [358269343-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a00c22c at consistency null
2022-03-22 05:58:18.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [358269343-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:19.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220550783-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a21872c at consistency null
2022-03-22 05:58:19.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220550783-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:19.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [585344589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b4291c3 at consistency null
2022-03-22 05:58:19.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [585344589-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:19.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [232842726-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@559fb336 at consistency null
2022-03-22 05:58:19.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [232842726-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:20.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324715004-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19c20a0b at consistency null
2022-03-22 05:58:20.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324715004-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:20.722 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:58:25.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1766111987-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f7cf6f8 at consistency null
2022-03-22 05:58:25.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1766111987-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:25.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1445636184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8fa45d at consistency null
2022-03-22 05:58:25.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1445636184-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:25.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1559210158-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2821b179 at consistency null
2022-03-22 05:58:25.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1559210158-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:26.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051161252-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21ba7424 at consistency null
2022-03-22 05:58:26.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051161252-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:26.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1920517281-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79298ddd at consistency null
2022-03-22 05:58:26.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1920517281-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:26.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1905511134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47963e9 at consistency null
2022-03-22 05:58:26.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1905511134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:26.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1352181402-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21f6a066 at consistency null
2022-03-22 05:58:26.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1352181402-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:26.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1437584511-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@531ffb28 at consistency null
2022-03-22 05:58:26.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1437584511-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:27.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1631999911-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@666dd3f3 at consistency null
2022-03-22 05:58:27.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1631999911-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:27.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250393707-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3da5f222 at consistency null
2022-03-22 05:58:27.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250393707-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:28.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466976602-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11cf8805 at consistency null
2022-03-22 05:58:28.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466976602-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:29.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808617067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19352384 at consistency null
2022-03-22 05:58:29.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808617067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:29.534 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:58:34.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662654117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14eb912c at consistency null
2022-03-22 05:58:34.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662654117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:34.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [146712897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cd5571 at consistency null
2022-03-22 05:58:34.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [146712897-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:34.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941697187-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6baf6a0d at consistency null
2022-03-22 05:58:34.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941697187-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:34.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479523344-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bb64dff at consistency null
2022-03-22 05:58:34.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479523344-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:35.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789078041-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@131cd554 at consistency null
2022-03-22 05:58:35.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789078041-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:35.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1375809560-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f967c9b at consistency null
2022-03-22 05:58:35.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1375809560-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:35.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674870475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f798e66 at consistency null
2022-03-22 05:58:35.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674870475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:36.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1483408595-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@649efc3 at consistency null
2022-03-22 05:58:36.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1483408595-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:36.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873447279-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@738a3c2b at consistency null
2022-03-22 05:58:36.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873447279-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:36.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1867896838-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c9d475b at consistency null
2022-03-22 05:58:36.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1867896838-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:37.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [800575745-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@556394fc at consistency null
2022-03-22 05:58:37.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [800575745-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:38.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [796530104-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77aa911f at consistency null
2022-03-22 05:58:38.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [796530104-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:38.105 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:58:41.481 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:58:41.484 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:58:43.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1715938303-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fa5c453 at consistency null
2022-03-22 05:58:43.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1715938303-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:43.167 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [655573721-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1974f163 at consistency null
2022-03-22 05:58:43.167 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [655573721-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:43.279 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [910008547-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50f942c5 at consistency null
2022-03-22 05:58:43.279 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [910008547-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:43.383 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1226155081-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@737803ac at consistency null
2022-03-22 05:58:43.383 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1226155081-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:43.540 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [378682959-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cffc611 at consistency null
2022-03-22 05:58:43.540 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [378682959-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:43.766 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724218830-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@168a4221 at consistency null
2022-03-22 05:58:43.766 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724218830-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:43.986 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1892361456-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25dc563f at consistency null
2022-03-22 05:58:43.987 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1892361456-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:44.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345567564-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61106cee at consistency null
2022-03-22 05:58:44.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345567564-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:45.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [327144918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@132df193 at consistency null
2022-03-22 05:58:45.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [327144918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:46.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1832235188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5772229d at consistency null
2022-03-22 05:58:46.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1832235188-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:46.911 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:58:51.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1156642118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1058fbe2 at consistency null
2022-03-22 05:58:51.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1156642118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:52.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389119493-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@724d821b at consistency null
2022-03-22 05:58:52.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389119493-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:52.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2056116304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e9054a8 at consistency null
2022-03-22 05:58:52.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2056116304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:52.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [989685669-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48b0e92d at consistency null
2022-03-22 05:58:52.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [989685669-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:52.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2036662137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f609c32 at consistency null
2022-03-22 05:58:52.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2036662137-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:52.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1706653446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a3889f at consistency null
2022-03-22 05:58:52.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1706653446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:52.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1902279865-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d9aa917 at consistency null
2022-03-22 05:58:52.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1902279865-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:53.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [544472447-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@422cd008 at consistency null
2022-03-22 05:58:53.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [544472447-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:53.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117222719-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59365573 at consistency null
2022-03-22 05:58:53.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117222719-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:54.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344080056-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44ccd9ba at consistency null
2022-03-22 05:58:54.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344080056-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:54.414 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:58:59.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855126454-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cc60225 at consistency null
2022-03-22 05:58:59.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855126454-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:59.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [224373395-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@384e3d25 at consistency null
2022-03-22 05:58:59.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [224373395-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:59.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1636313534-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53b86a92 at consistency null
2022-03-22 05:58:59.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1636313534-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:58:59.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1500030463-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@711b6856 at consistency null
2022-03-22 05:58:59.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1500030463-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:00.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720263909-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5de08b7a at consistency null
2022-03-22 05:59:00.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720263909-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:00.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1511355929-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@140e22f6 at consistency null
2022-03-22 05:59:00.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1511355929-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:01.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054185276-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@472261f at consistency null
2022-03-22 05:59:01.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054185276-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:02.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1714119888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e01a71f at consistency null
2022-03-22 05:59:02.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1714119888-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:02.212 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:59:07.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1053438850-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@776342fe at consistency null
2022-03-22 05:59:07.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1053438850-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:07.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1335767036-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@129fd776 at consistency null
2022-03-22 05:59:07.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1335767036-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:07.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916377266-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74849e4 at consistency null
2022-03-22 05:59:07.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916377266-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:07.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [187576661-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24523f4d at consistency null
2022-03-22 05:59:07.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [187576661-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:07.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1679246737-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49cbed8b at consistency null
2022-03-22 05:59:07.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1679246737-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:07.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523083040-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2503a7a3 at consistency null
2022-03-22 05:59:07.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523083040-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:07.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [442751381-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60c1dd8e at consistency null
2022-03-22 05:59:07.959 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [442751381-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:08.374 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1623058197-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@555c2e8c at consistency null
2022-03-22 05:59:08.374 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1623058197-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:09.182 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1045962078-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43ed3619 at consistency null
2022-03-22 05:59:09.182 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1045962078-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:10.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [665885906-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b6bbf78 at consistency null
2022-03-22 05:59:10.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [665885906-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:10.830 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:59:11.513 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:59:11.536 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:59:15.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [588994240-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15282036 at consistency null
2022-03-22 05:59:15.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [588994240-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:15.893 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [739346681-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21b45451 at consistency null
2022-03-22 05:59:15.893 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [739346681-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:15.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1390269128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bc3f380 at consistency null
2022-03-22 05:59:15.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1390269128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:15.986 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1906868813-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2df51dff at consistency null
2022-03-22 05:59:15.987 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1906868813-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:16.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142374894-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7908b1e7 at consistency null
2022-03-22 05:59:16.047 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142374894-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:16.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680157272-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a9d744b at consistency null
2022-03-22 05:59:16.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680157272-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:16.160 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [343780371-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ed6c76b at consistency null
2022-03-22 05:59:16.160 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [343780371-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:16.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [569572923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bc925ab at consistency null
2022-03-22 05:59:16.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [569572923-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:16.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1255816940-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@577236bc at consistency null
2022-03-22 05:59:16.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1255816940-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:16.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [833163087-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@465c3f35 at consistency null
2022-03-22 05:59:16.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [833163087-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:17.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213014109-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ee9550a at consistency null
2022-03-22 05:59:17.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213014109-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:17.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2130189385-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cafdf01 at consistency null
2022-03-22 05:59:17.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2130189385-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:19.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1708023844-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a2ec9fe at consistency null
2022-03-22 05:59:19.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1708023844-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:19.510 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:59:24.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [476367013-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dacf419 at consistency null
2022-03-22 05:59:24.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [476367013-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:24.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1032473788-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@939bba3 at consistency null
2022-03-22 05:59:24.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1032473788-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:24.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [53351696-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b2c2a52 at consistency null
2022-03-22 05:59:24.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [53351696-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:24.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [368548369-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47416bf0 at consistency null
2022-03-22 05:59:24.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [368548369-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:25.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804018666-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d77433d at consistency null
2022-03-22 05:59:25.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804018666-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:25.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1089216263-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39d71b93 at consistency null
2022-03-22 05:59:25.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1089216263-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:25.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [857281695-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1303c841 at consistency null
2022-03-22 05:59:25.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [857281695-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:26.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [446117887-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fd13f3a at consistency null
2022-03-22 05:59:26.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [446117887-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:26.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475944014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18f8e156 at consistency null
2022-03-22 05:59:26.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475944014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:27.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2001545105-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19f27a59 at consistency null
2022-03-22 05:59:27.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2001545105-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:27.711 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:59:32.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [210462731-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14bee8e6 at consistency null
2022-03-22 05:59:32.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [210462731-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:32.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [788569770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42c03da4 at consistency null
2022-03-22 05:59:32.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [788569770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:33.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [815070826-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d6d86fb at consistency null
2022-03-22 05:59:33.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [815070826-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:33.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779826758-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@174707bd at consistency null
2022-03-22 05:59:33.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779826758-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:33.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2036844722-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74dd346b at consistency null
2022-03-22 05:59:33.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2036844722-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:33.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1387916538-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37331a49 at consistency null
2022-03-22 05:59:33.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1387916538-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:34.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1170816873-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4698180a at consistency null
2022-03-22 05:59:34.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1170816873-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:35.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855107728-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@694d958b at consistency null
2022-03-22 05:59:35.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855107728-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:36.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1964151844-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@742e8aa5 at consistency null
2022-03-22 05:59:36.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1964151844-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:36.260 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:59:41.269 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1758554176-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b60373b at consistency null
2022-03-22 05:59:41.269 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1758554176-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:41.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1685008452-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d3e3c15 at consistency null
2022-03-22 05:59:41.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1685008452-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:41.388 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1766975281-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58da0bf4 at consistency null
2022-03-22 05:59:41.388 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1766975281-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:41.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142867851-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58a1c9e8 at consistency null
2022-03-22 05:59:41.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [142867851-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:41.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1802704819-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a2e428 at consistency null
2022-03-22 05:59:41.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1802704819-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:41.541 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 05:59:41.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 05:59:41.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1988360073-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e50aefd at consistency null
2022-03-22 05:59:41.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1988360073-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:41.764 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [277736967-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26dd8568 at consistency null
2022-03-22 05:59:41.764 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [277736967-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:41.992 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [125122104-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70e5e6d2 at consistency null
2022-03-22 05:59:41.992 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [125122104-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:42.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959294954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6573df42 at consistency null
2022-03-22 05:59:42.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959294954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:42.448 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [488569157-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4effa257 at consistency null
2022-03-22 05:59:42.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [488569157-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:42.855 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1646051639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c39825 at consistency null
2022-03-22 05:59:42.855 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1646051639-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:43.272 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282867130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bcf2d19 at consistency null
2022-03-22 05:59:43.272 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1282867130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:43.699 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2067183353-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@203d60ed at consistency null
2022-03-22 05:59:43.699 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2067183353-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:44.531 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [294169611-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@136393a7 at consistency null
2022-03-22 05:59:44.531 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [294169611-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:44.533 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:59:49.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [354013812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bf9335e at consistency null
2022-03-22 05:59:49.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [354013812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:49.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860793410-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@297c70dd at consistency null
2022-03-22 05:59:49.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860793410-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:50.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [539395608-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@628d4717 at consistency null
2022-03-22 05:59:50.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [539395608-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:50.300 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1947901667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6dfc0e4a at consistency null
2022-03-22 05:59:50.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1947901667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:50.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [793475174-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f5da368 at consistency null
2022-03-22 05:59:50.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [793475174-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:51.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [170218153-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@656dd4c6 at consistency null
2022-03-22 05:59:51.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [170218153-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:51.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [505873331-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d509283 at consistency null
2022-03-22 05:59:51.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [505873331-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:52.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1055550226-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60535f3d at consistency null
2022-03-22 05:59:52.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1055550226-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:53.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243000033-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2257f6bc at consistency null
2022-03-22 05:59:53.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243000033-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:53.029 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 05:59:58.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1787170576-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2997ddd4 at consistency null
2022-03-22 05:59:58.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1787170576-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:58.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1285202792-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@403a96ef at consistency null
2022-03-22 05:59:58.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1285202792-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:58.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [558228242-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a6a8362 at consistency null
2022-03-22 05:59:58.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [558228242-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:58.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1590375039-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27156085 at consistency null
2022-03-22 05:59:58.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1590375039-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:58.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74405945-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30fb60ce at consistency null
2022-03-22 05:59:58.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74405945-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:58.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [494348863-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24e215d0 at consistency null
2022-03-22 05:59:58.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [494348863-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:58.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [321128432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a7df3bd at consistency null
2022-03-22 05:59:58.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [321128432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:58.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [352032545-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46e0b33f at consistency null
2022-03-22 05:59:58.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [352032545-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:59.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1789639447-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d76563d at consistency null
2022-03-22 05:59:59.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1789639447-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 05:59:59.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1311326895-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@190e8ad6 at consistency null
2022-03-22 05:59:59.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1311326895-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:00.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2097268509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ccb7251 at consistency null
2022-03-22 06:00:00.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2097268509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:01.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889914627-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ff147bc at consistency null
2022-03-22 06:00:01.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889914627-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:01.304 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:00:06.358 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [38757302-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60a931c1 at consistency null
2022-03-22 06:00:06.359 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [38757302-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:06.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [116880746-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d9c4f38 at consistency null
2022-03-22 06:00:06.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [116880746-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:06.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1966957446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6aa2fd20 at consistency null
2022-03-22 06:00:06.544 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1966957446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:06.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [418595647-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f9b8b6f at consistency null
2022-03-22 06:00:06.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [418595647-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:06.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1633206596-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36159664 at consistency null
2022-03-22 06:00:06.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1633206596-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:07.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555670048-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@599fea11 at consistency null
2022-03-22 06:00:07.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555670048-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:07.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1116134754-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@242947ff at consistency null
2022-03-22 06:00:07.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1116134754-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:08.248 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1089373366-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@672a70e6 at consistency null
2022-03-22 06:00:08.248 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1089373366-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:09.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1738223374-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d967cd1 at consistency null
2022-03-22 06:00:09.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1738223374-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:09.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989283705-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b7fce10 at consistency null
2022-03-22 06:00:09.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989283705-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:09.906 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:00:11.573 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:00:11.578 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:00:14.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [42705156-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35077c3f at consistency null
2022-03-22 06:00:14.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [42705156-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:15.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921984727-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11e8739 at consistency null
2022-03-22 06:00:15.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921984727-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:15.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136967256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f20033f at consistency null
2022-03-22 06:00:15.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136967256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:15.359 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1848036323-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a0a4b85 at consistency null
2022-03-22 06:00:15.359 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1848036323-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:15.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [838517922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6462247f at consistency null
2022-03-22 06:00:15.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [838517922-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:15.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636233388-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11b05085 at consistency null
2022-03-22 06:00:15.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636233388-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:16.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1654693245-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ec69077 at consistency null
2022-03-22 06:00:16.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1654693245-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:16.468 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981543705-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66a8ae49 at consistency null
2022-03-22 06:00:16.468 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981543705-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:16.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572419391-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@131c3f3f at consistency null
2022-03-22 06:00:16.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572419391-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:17.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421831522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bb61a1 at consistency null
2022-03-22 06:00:17.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421831522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:18.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2139725390-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a940ce9 at consistency null
2022-03-22 06:00:18.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2139725390-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:18.217 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:00:23.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1289993029-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@496d3d14 at consistency null
2022-03-22 06:00:23.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1289993029-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:23.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484225833-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bfa16c1 at consistency null
2022-03-22 06:00:23.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484225833-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:23.552 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1685383926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6097ce76 at consistency null
2022-03-22 06:00:23.552 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1685383926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:23.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1471964120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2aae65c9 at consistency null
2022-03-22 06:00:23.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1471964120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:23.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [114607097-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a0b7815 at consistency null
2022-03-22 06:00:23.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [114607097-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:24.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1697212479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61ac2b8b at consistency null
2022-03-22 06:00:24.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1697212479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:24.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [642325814-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f16adb6 at consistency null
2022-03-22 06:00:24.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [642325814-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:24.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [308222084-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66cf17b0 at consistency null
2022-03-22 06:00:24.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [308222084-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:24.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [842320445-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@375510ac at consistency null
2022-03-22 06:00:24.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [842320445-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:25.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [477246676-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@372585be at consistency null
2022-03-22 06:00:25.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [477246676-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:25.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1924938433-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d54e5f1 at consistency null
2022-03-22 06:00:25.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1924938433-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:25.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407738154-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23b8077c at consistency null
2022-03-22 06:00:25.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407738154-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:25.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [213701436-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c77b3d7 at consistency null
2022-03-22 06:00:25.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [213701436-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:26.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [31818475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2764d992 at consistency null
2022-03-22 06:00:26.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [31818475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:26.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [303880740-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e65b8c9 at consistency null
2022-03-22 06:00:26.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [303880740-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:27.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721564042-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f8d51d9 at consistency null
2022-03-22 06:00:27.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721564042-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:27.224 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:00:32.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [215216470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ea4bbf3 at consistency null
2022-03-22 06:00:32.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [215216470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:32.435 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [554486013-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ded310e at consistency null
2022-03-22 06:00:32.435 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [554486013-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:32.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1437985538-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e3d7b8a at consistency null
2022-03-22 06:00:32.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1437985538-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:32.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1425283733-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b0bf60d at consistency null
2022-03-22 06:00:32.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1425283733-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:32.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1203640348-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@708bbfbc at consistency null
2022-03-22 06:00:32.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1203640348-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:32.848 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [3304246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@544a95e2 at consistency null
2022-03-22 06:00:32.848 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [3304246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:33.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2057665993-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fe45ca at consistency null
2022-03-22 06:00:33.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2057665993-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:33.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250732730-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55917066 at consistency null
2022-03-22 06:00:33.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [250732730-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:33.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [968386432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78ca5337 at consistency null
2022-03-22 06:00:33.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [968386432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:33.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [708890714-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78af3c49 at consistency null
2022-03-22 06:00:33.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [708890714-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:34.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2069543918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b0ce683 at consistency null
2022-03-22 06:00:34.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2069543918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:35.078 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049925135-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55c0a5c4 at consistency null
2022-03-22 06:00:35.079 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049925135-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:35.080 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:00:40.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5344429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fb917a0 at consistency null
2022-03-22 06:00:40.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5344429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:40.249 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [742350589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e75216a at consistency null
2022-03-22 06:00:40.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [742350589-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:40.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43259231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12a1f8d9 at consistency null
2022-03-22 06:00:40.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43259231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:40.388 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [62530074-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@421f91cc at consistency null
2022-03-22 06:00:40.388 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [62530074-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:40.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [260239908-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22859b96 at consistency null
2022-03-22 06:00:40.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [260239908-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:40.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1503432095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fffd82f at consistency null
2022-03-22 06:00:40.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1503432095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:40.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063615262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d0afed at consistency null
2022-03-22 06:00:40.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063615262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:41.138 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695734934-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10bc18a5 at consistency null
2022-03-22 06:00:41.138 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695734934-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:41.347 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [829670991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e6b780f at consistency null
2022-03-22 06:00:41.347 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [829670991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:41.601 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:00:41.610 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:00:41.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [213800331-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f14e54d at consistency null
2022-03-22 06:00:41.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [213800331-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:42.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1978653111-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b3f8685 at consistency null
2022-03-22 06:00:42.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1978653111-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:43.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [476743998-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b36d477 at consistency null
2022-03-22 06:00:43.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [476743998-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:43.443 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:00:48.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855926082-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ed86a1e at consistency null
2022-03-22 06:00:48.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855926082-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:48.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298689188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@161ec405 at consistency null
2022-03-22 06:00:48.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298689188-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:48.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1187067615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@313263ed at consistency null
2022-03-22 06:00:48.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1187067615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:48.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [237659200-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cba9cd0 at consistency null
2022-03-22 06:00:48.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [237659200-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:48.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [513027762-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11f8bc3 at consistency null
2022-03-22 06:00:48.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [513027762-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:49.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269970568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61c15909 at consistency null
2022-03-22 06:00:49.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269970568-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:49.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596966811-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@434e63d7 at consistency null
2022-03-22 06:00:49.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596966811-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:49.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49983578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4656ee5f at consistency null
2022-03-22 06:00:49.539 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49983578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:49.774 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1034219273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2db4fe8d at consistency null
2022-03-22 06:00:49.774 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1034219273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:50.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [385656601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dd46037 at consistency null
2022-03-22 06:00:50.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [385656601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:51.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [927906114-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@778f7944 at consistency null
2022-03-22 06:00:51.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [927906114-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:52.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1625401233-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65a45fc at consistency null
2022-03-22 06:00:52.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1625401233-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:52.028 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:00:57.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2057814051-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c10832c at consistency null
2022-03-22 06:00:57.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2057814051-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:57.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [152069121-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32fb9f41 at consistency null
2022-03-22 06:00:57.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [152069121-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:57.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1272334536-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3eb29503 at consistency null
2022-03-22 06:00:57.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1272334536-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:57.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [652653324-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a93528c at consistency null
2022-03-22 06:00:57.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [652653324-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:58.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644484907-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6292a511 at consistency null
2022-03-22 06:00:58.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644484907-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:58.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757321478-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cfd2f95 at consistency null
2022-03-22 06:00:58.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757321478-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:59.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299306193-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26cda984 at consistency null
2022-03-22 06:00:59.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299306193-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:00:59.814 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:01:04.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [37401396-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56ba88e3 at consistency null
2022-03-22 06:01:04.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [37401396-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:04.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1177812953-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21770508 at consistency null
2022-03-22 06:01:04.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1177812953-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:05.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524185264-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44931734 at consistency null
2022-03-22 06:01:05.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524185264-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:05.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1933641406-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fd907ab at consistency null
2022-03-22 06:01:05.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1933641406-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:05.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607500529-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@146070ca at consistency null
2022-03-22 06:01:05.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607500529-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:06.556 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2107725465-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4816efc5 at consistency null
2022-03-22 06:01:06.556 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2107725465-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:07.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1848569125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f355d33 at consistency null
2022-03-22 06:01:07.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1848569125-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:07.404 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:01:11.631 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:01:11.641 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:01:12.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [354509480-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31744085 at consistency null
2022-03-22 06:01:12.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [354509480-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:12.480 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484090573-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26859041 at consistency null
2022-03-22 06:01:12.480 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484090573-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:12.591 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1798842961-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59122b93 at consistency null
2022-03-22 06:01:12.591 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1798842961-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:12.699 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919333846-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ceb3947 at consistency null
2022-03-22 06:01:12.699 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919333846-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:12.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1816968464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f882e37 at consistency null
2022-03-22 06:01:12.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1816968464-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:12.957 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [625583774-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6505bc1f at consistency null
2022-03-22 06:01:12.957 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [625583774-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:13.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [252275396-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d835d9f at consistency null
2022-03-22 06:01:13.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [252275396-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:13.270 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1208395510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8877869 at consistency null
2022-03-22 06:01:13.271 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1208395510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:13.686 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1093347035-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@265016d8 at consistency null
2022-03-22 06:01:13.686 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1093347035-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:14.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [691930422-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e7db349 at consistency null
2022-03-22 06:01:14.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [691930422-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:14.957 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1831146911-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@592564bc at consistency null
2022-03-22 06:01:14.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1831146911-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:14.959 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:01:20.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [283721691-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a520bc1 at consistency null
2022-03-22 06:01:20.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [283721691-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:20.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [149737187-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c122685 at consistency null
2022-03-22 06:01:20.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [149737187-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:20.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1361214845-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ff633b5 at consistency null
2022-03-22 06:01:20.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1361214845-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:20.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [351146795-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d04be73 at consistency null
2022-03-22 06:01:20.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [351146795-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:20.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294000522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4772a611 at consistency null
2022-03-22 06:01:20.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294000522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:20.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [592819483-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a899674 at consistency null
2022-03-22 06:01:20.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [592819483-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:20.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1641999769-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@641c650b at consistency null
2022-03-22 06:01:20.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1641999769-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:21.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [97377730-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c73a8dd at consistency null
2022-03-22 06:01:21.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [97377730-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:21.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1328234560-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e63cb6c at consistency null
2022-03-22 06:01:21.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1328234560-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:23.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096337875-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@780fc2e2 at consistency null
2022-03-22 06:01:23.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096337875-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:23.228 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:01:28.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1467766340-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f3cc4d5 at consistency null
2022-03-22 06:01:28.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1467766340-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:28.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1010082468-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ee8beaf at consistency null
2022-03-22 06:01:28.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1010082468-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:28.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [696835106-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ee03eb5 at consistency null
2022-03-22 06:01:28.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [696835106-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:28.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [773430929-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28fee159 at consistency null
2022-03-22 06:01:28.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [773430929-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:29.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [467809120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@317d82ff at consistency null
2022-03-22 06:01:29.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [467809120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:29.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47874874-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6abb5fb9 at consistency null
2022-03-22 06:01:29.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47874874-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:29.740 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1843148431-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4501c1cd at consistency null
2022-03-22 06:01:29.740 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1843148431-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:30.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257151155-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1aaa141d at consistency null
2022-03-22 06:01:30.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257151155-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:31.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [94407434-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@471d07cd at consistency null
2022-03-22 06:01:31.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [94407434-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:31.639 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:01:36.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1395989463-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@776bce00 at consistency null
2022-03-22 06:01:36.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1395989463-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:36.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599158128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c2f7689 at consistency null
2022-03-22 06:01:36.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599158128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:36.988 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318494817-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@766f97ed at consistency null
2022-03-22 06:01:36.988 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318494817-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:37.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1332709474-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5963c582 at consistency null
2022-03-22 06:01:37.243 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1332709474-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:37.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [880790011-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@378da0cd at consistency null
2022-03-22 06:01:37.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [880790011-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:38.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1648591539-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dd7992e at consistency null
2022-03-22 06:01:38.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1648591539-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:40.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [423469360-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a5f04c3 at consistency null
2022-03-22 06:01:40.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [423469360-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:40.216 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:01:41.661 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:01:41.720 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:01:45.244 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [706873069-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22d9ae3 at consistency null
2022-03-22 06:01:45.244 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [706873069-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:45.284 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1616392771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@587cd9a3 at consistency null
2022-03-22 06:01:45.284 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1616392771-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:45.364 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510681825-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59e3356b at consistency null
2022-03-22 06:01:45.364 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510681825-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:45.482 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1781556286-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b985b88 at consistency null
2022-03-22 06:01:45.482 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1781556286-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:45.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1352598267-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@350c289a at consistency null
2022-03-22 06:01:45.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1352598267-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:45.967 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903487105-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b835e59 at consistency null
2022-03-22 06:01:45.967 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903487105-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:46.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1812917028-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b8b7a2f at consistency null
2022-03-22 06:01:46.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1812917028-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:46.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [91644687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52562ccd at consistency null
2022-03-22 06:01:46.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [91644687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:47.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [760623488-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64a86e5 at consistency null
2022-03-22 06:01:47.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [760623488-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:48.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2075194443-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@172a71e3 at consistency null
2022-03-22 06:01:48.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2075194443-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:49.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1627154386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7603b2b at consistency null
2022-03-22 06:01:49.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1627154386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:49.233 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:01:54.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1445207769-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64335f4e at consistency null
2022-03-22 06:01:54.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1445207769-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:54.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256386476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@369c9083 at consistency null
2022-03-22 06:01:54.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256386476-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:54.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906774551-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72fd5ada at consistency null
2022-03-22 06:01:54.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906774551-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:54.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [894270451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74c29ecd at consistency null
2022-03-22 06:01:54.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [894270451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:54.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1130981717-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a757634 at consistency null
2022-03-22 06:01:54.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1130981717-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:55.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [506163483-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58bfe771 at consistency null
2022-03-22 06:01:55.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [506163483-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:55.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1244913209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@408c5d36 at consistency null
2022-03-22 06:01:55.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1244913209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:55.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [343914095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20d8871c at consistency null
2022-03-22 06:01:55.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [343914095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:56.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475289204-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bd81e5 at consistency null
2022-03-22 06:01:56.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [475289204-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:01:56.735 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:02:01.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1496473420-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@261e6acc at consistency null
2022-03-22 06:02:01.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1496473420-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:01.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [228888645-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d9e6d4 at consistency null
2022-03-22 06:02:01.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [228888645-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:02.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1686209360-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@605d7134 at consistency null
2022-03-22 06:02:02.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1686209360-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:02.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524145730-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d027673 at consistency null
2022-03-22 06:02:02.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524145730-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:02.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [291205738-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@589f3515 at consistency null
2022-03-22 06:02:02.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [291205738-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:03.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [953482940-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@133b8794 at consistency null
2022-03-22 06:02:03.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [953482940-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:03.895 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344758261-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18e49016 at consistency null
2022-03-22 06:02:03.895 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344758261-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:04.752 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1309921085-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64bdc97 at consistency null
2022-03-22 06:02:04.752 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1309921085-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:04.763 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:02:09.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [76623031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@275d66d6 at consistency null
2022-03-22 06:02:09.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [76623031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:09.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [456070023-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6815528e at consistency null
2022-03-22 06:02:09.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [456070023-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:10.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886183902-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54cd0bba at consistency null
2022-03-22 06:02:10.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886183902-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:10.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550941385-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cbacad0 at consistency null
2022-03-22 06:02:10.234 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550941385-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:10.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [766498544-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@409d6647 at consistency null
2022-03-22 06:02:10.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [766498544-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:10.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1325358057-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25eddf8 at consistency null
2022-03-22 06:02:10.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1325358057-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:10.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1493836281-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ef286d2 at consistency null
2022-03-22 06:02:10.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1493836281-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:11.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1289715947-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64443796 at consistency null
2022-03-22 06:02:11.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1289715947-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:11.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [521455574-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fc0da07 at consistency null
2022-03-22 06:02:11.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [521455574-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:11.545 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [729868238-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b056f1a at consistency null
2022-03-22 06:02:11.545 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [729868238-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:11.721 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:02:11.739 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:02:11.952 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [588058286-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11ebaabe at consistency null
2022-03-22 06:02:11.952 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [588058286-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:12.765 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [325393148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54d6c792 at consistency null
2022-03-22 06:02:12.766 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [325393148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:13.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [712394033-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d0f3fed at consistency null
2022-03-22 06:02:13.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [712394033-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:13.628 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:02:18.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [115632422-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@de2eee9 at consistency null
2022-03-22 06:02:18.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [115632422-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:18.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1297575722-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41900bb1 at consistency null
2022-03-22 06:02:18.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1297575722-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:18.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1994276486-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2398ae46 at consistency null
2022-03-22 06:02:18.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1994276486-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:19.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1847177889-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@478c0d8f at consistency null
2022-03-22 06:02:19.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1847177889-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:19.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267813149-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d032631 at consistency null
2022-03-22 06:02:19.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267813149-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:20.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1036604282-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c8f73a3 at consistency null
2022-03-22 06:02:20.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1036604282-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:21.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [988049483-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a8314ea at consistency null
2022-03-22 06:02:21.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [988049483-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:22.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [432750860-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dc48e40 at consistency null
2022-03-22 06:02:22.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [432750860-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:22.017 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:02:27.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1489189003-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19320967 at consistency null
2022-03-22 06:02:27.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1489189003-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:27.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [803660394-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ec77389 at consistency null
2022-03-22 06:02:27.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [803660394-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:27.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916040181-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c810cff at consistency null
2022-03-22 06:02:27.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [916040181-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:27.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1972981003-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65ce27a1 at consistency null
2022-03-22 06:02:27.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1972981003-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:28.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [887043155-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c3e634e at consistency null
2022-03-22 06:02:28.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [887043155-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:28.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [741500176-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29a01e7b at consistency null
2022-03-22 06:02:28.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [741500176-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:29.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1461692258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dd20944 at consistency null
2022-03-22 06:02:29.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1461692258-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:30.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1758546335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2935c270 at consistency null
2022-03-22 06:02:30.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1758546335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:30.234 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:02:35.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [635540858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f746377 at consistency null
2022-03-22 06:02:35.341 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [635540858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:35.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [449080883-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32c1c1f4 at consistency null
2022-03-22 06:02:35.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [449080883-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:35.561 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [810221246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35e0b39e at consistency null
2022-03-22 06:02:35.562 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [810221246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:35.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1379255449-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@742f1391 at consistency null
2022-03-22 06:02:35.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1379255449-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:35.794 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2069283296-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@487ca9b8 at consistency null
2022-03-22 06:02:35.795 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2069283296-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:35.989 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1121619279-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@740e0f22 at consistency null
2022-03-22 06:02:35.989 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1121619279-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:36.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629290148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44c22e7e at consistency null
2022-03-22 06:02:36.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629290148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:36.494 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1500462465-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@487d0eec at consistency null
2022-03-22 06:02:36.494 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1500462465-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:36.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [352038541-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65e15f32 at consistency null
2022-03-22 06:02:36.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [352038541-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:37.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [48101800-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61122bb3 at consistency null
2022-03-22 06:02:37.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [48101800-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:38.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [911133251-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34f64831 at consistency null
2022-03-22 06:02:38.041 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [911133251-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:38.054 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:02:41.749 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:02:41.804 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:02:43.063 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [384657982-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@145c2a2d at consistency null
2022-03-22 06:02:43.063 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [384657982-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:43.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596666280-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7babef5b at consistency null
2022-03-22 06:02:43.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596666280-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:43.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [852919132-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b141c2e at consistency null
2022-03-22 06:02:43.233 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [852919132-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:43.349 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1913277335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d257abd at consistency null
2022-03-22 06:02:43.349 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1913277335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:43.581 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256602994-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@785d3476 at consistency null
2022-03-22 06:02:43.581 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256602994-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:43.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [158049602-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ca8dd11 at consistency null
2022-03-22 06:02:43.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [158049602-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:44.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [253827920-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77764fc6 at consistency null
2022-03-22 06:02:44.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [253827920-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:44.440 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [802668496-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51a02307 at consistency null
2022-03-22 06:02:44.440 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [802668496-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:44.894 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2069076411-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40cf53c1 at consistency null
2022-03-22 06:02:44.894 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2069076411-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:45.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [801534502-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a955e9c at consistency null
2022-03-22 06:02:45.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [801534502-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:45.708 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:02:50.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674895637-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f583277 at consistency null
2022-03-22 06:02:50.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674895637-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:50.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [973145341-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fb09280 at consistency null
2022-03-22 06:02:50.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [973145341-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:50.951 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576590407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@755b91ec at consistency null
2022-03-22 06:02:50.951 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576590407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:51.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1102299804-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54a5065 at consistency null
2022-03-22 06:02:51.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1102299804-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:51.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969537280-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b5f0131 at consistency null
2022-03-22 06:02:51.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969537280-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:51.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199043624-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10b0bc48 at consistency null
2022-03-22 06:02:51.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199043624-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:51.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [274515440-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@394d8b43 at consistency null
2022-03-22 06:02:51.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [274515440-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:51.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1327776325-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e532d77 at consistency null
2022-03-22 06:02:51.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1327776325-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:52.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [967040342-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4da6ff05 at consistency null
2022-03-22 06:02:52.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [967040342-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:52.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531788320-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58e14805 at consistency null
2022-03-22 06:02:52.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531788320-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:52.752 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768082428-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d23d7c9 at consistency null
2022-03-22 06:02:52.753 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768082428-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:53.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1075005546-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7aec7b8f at consistency null
2022-03-22 06:02:53.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1075005546-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:54.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736270931-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48ed00 at consistency null
2022-03-22 06:02:54.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736270931-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:54.031 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:02:59.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1441003680-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@523b2c6c at consistency null
2022-03-22 06:02:59.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1441003680-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:59.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [952751105-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41b5347d at consistency null
2022-03-22 06:02:59.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [952751105-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:59.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1648593770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@581c1fc7 at consistency null
2022-03-22 06:02:59.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1648593770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:59.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1854940301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f7bd929 at consistency null
2022-03-22 06:02:59.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1854940301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:02:59.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [620553991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1428cee6 at consistency null
2022-03-22 06:02:59.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [620553991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:00.140 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1382237765-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c5606c2 at consistency null
2022-03-22 06:03:00.140 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1382237765-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:00.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [738561976-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@440ccca8 at consistency null
2022-03-22 06:03:00.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [738561976-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:00.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138651897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57ff791 at consistency null
2022-03-22 06:03:00.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1138651897-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:01.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5857626-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57725626 at consistency null
2022-03-22 06:03:01.359 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5857626-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:02.176 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1212178042-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bde2a1c at consistency null
2022-03-22 06:03:02.176 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1212178042-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:02.178 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:03:07.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211654277-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3915dc43 at consistency null
2022-03-22 06:03:07.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211654277-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:07.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025953985-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@432c678f at consistency null
2022-03-22 06:03:07.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025953985-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:07.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [861497138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cce9eb9 at consistency null
2022-03-22 06:03:07.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [861497138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:07.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040552621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7baea0a7 at consistency null
2022-03-22 06:03:07.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040552621-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:07.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1914120020-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b1eb042 at consistency null
2022-03-22 06:03:07.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1914120020-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:07.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [116702277-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bb1ce01 at consistency null
2022-03-22 06:03:07.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [116702277-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:08.034 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2121018206-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1392a298 at consistency null
2022-03-22 06:03:08.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2121018206-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:08.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [89277706-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1db793d5 at consistency null
2022-03-22 06:03:08.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [89277706-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:08.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1312096115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3385bc81 at consistency null
2022-03-22 06:03:08.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1312096115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:08.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1837196388-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36c1bd36 at consistency null
2022-03-22 06:03:08.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1837196388-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:09.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1915257937-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30c14f14 at consistency null
2022-03-22 06:03:09.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1915257937-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:09.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1699083531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11c95be2 at consistency null
2022-03-22 06:03:09.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1699083531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:10.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [216220087-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fb68cee at consistency null
2022-03-22 06:03:10.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [216220087-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:10.331 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:03:11.806 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:03:11.810 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:03:15.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1780987403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3425a823 at consistency null
2022-03-22 06:03:15.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1780987403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:15.400 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553706258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55eba421 at consistency null
2022-03-22 06:03:15.400 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553706258-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:15.455 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [381478439-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e81938e at consistency null
2022-03-22 06:03:15.455 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [381478439-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:15.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12903358-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@777c4acd at consistency null
2022-03-22 06:03:15.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12903358-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:15.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1468308035-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e21b876 at consistency null
2022-03-22 06:03:15.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1468308035-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:15.846 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1458982556-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4663d73f at consistency null
2022-03-22 06:03:15.846 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1458982556-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:16.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [654972859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@241c319e at consistency null
2022-03-22 06:03:16.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [654972859-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:16.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [16046583-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31232df2 at consistency null
2022-03-22 06:03:16.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [16046583-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:17.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1934691624-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7282d1a8 at consistency null
2022-03-22 06:03:17.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1934691624-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:17.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389496688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a803ae8 at consistency null
2022-03-22 06:03:17.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389496688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:17.925 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:03:23.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626818018-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f556ee7 at consistency null
2022-03-22 06:03:23.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626818018-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:23.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181356906-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47b19242 at consistency null
2022-03-22 06:03:23.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181356906-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:23.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026129602-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@235a148d at consistency null
2022-03-22 06:03:23.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026129602-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:23.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [323689128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8263d74 at consistency null
2022-03-22 06:03:23.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [323689128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:23.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1229559158-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cc048c1 at consistency null
2022-03-22 06:03:23.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1229559158-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:23.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596866166-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a8f099 at consistency null
2022-03-22 06:03:23.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596866166-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:24.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [688358463-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74a5629d at consistency null
2022-03-22 06:03:24.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [688358463-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:24.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1657951725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d513ce0 at consistency null
2022-03-22 06:03:24.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1657951725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:25.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1739951844-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@319f3cb at consistency null
2022-03-22 06:03:25.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1739951844-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:25.657 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1229378026-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43f5f709 at consistency null
2022-03-22 06:03:25.657 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1229378026-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:26.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1559593601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c1bc460 at consistency null
2022-03-22 06:03:26.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1559593601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:26.505 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:03:31.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754506191-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58694d3f at consistency null
2022-03-22 06:03:31.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754506191-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:31.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1743888410-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14a27a52 at consistency null
2022-03-22 06:03:31.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1743888410-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:31.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407364751-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78039a70 at consistency null
2022-03-22 06:03:31.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407364751-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:31.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1758891148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3af969d4 at consistency null
2022-03-22 06:03:31.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1758891148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:31.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [42604280-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@286bcb01 at consistency null
2022-03-22 06:03:31.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [42604280-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:32.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754372035-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f9c54b4 at consistency null
2022-03-22 06:03:32.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754372035-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:32.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1339389378-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b288b2d at consistency null
2022-03-22 06:03:32.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1339389378-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:32.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [331035224-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3aeeec55 at consistency null
2022-03-22 06:03:32.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [331035224-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:32.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1104468737-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cab0bca at consistency null
2022-03-22 06:03:32.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1104468737-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:32.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [631206775-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dfdf1f at consistency null
2022-03-22 06:03:32.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [631206775-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:32.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098934627-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e215dea at consistency null
2022-03-22 06:03:32.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098934627-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:32.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [191085305-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4360943f at consistency null
2022-03-22 06:03:32.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [191085305-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:33.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820881991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@570122c2 at consistency null
2022-03-22 06:03:33.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820881991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:33.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [844281450-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2859281a at consistency null
2022-03-22 06:03:33.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [844281450-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:33.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893407282-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c93eef6 at consistency null
2022-03-22 06:03:33.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893407282-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:33.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727644322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b954637 at consistency null
2022-03-22 06:03:33.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727644322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:34.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1241294966-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@542489ec at consistency null
2022-03-22 06:03:34.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1241294966-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:35.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1849337541-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@677339c0 at consistency null
2022-03-22 06:03:35.034 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1849337541-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:35.035 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:03:40.077 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175989178-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70e99456 at consistency null
2022-03-22 06:03:40.077 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175989178-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:40.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851530505-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e319ca9 at consistency null
2022-03-22 06:03:40.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851530505-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:40.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112516524-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b4dadcf at consistency null
2022-03-22 06:03:40.247 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112516524-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:40.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046307758-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57badfd4 at consistency null
2022-03-22 06:03:40.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046307758-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:40.469 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117766352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@471a1d15 at consistency null
2022-03-22 06:03:40.469 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117766352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:40.589 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887461963-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35bc158b at consistency null
2022-03-22 06:03:40.589 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887461963-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:40.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1338890544-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3319e526 at consistency null
2022-03-22 06:03:40.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1338890544-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:41.247 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2002786362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32dcc8ee at consistency null
2022-03-22 06:03:41.247 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2002786362-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:41.669 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1570129418-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25b98915 at consistency null
2022-03-22 06:03:41.669 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1570129418-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:41.837 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:03:41.856 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:03:42.076 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779613641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@707cc064 at consistency null
2022-03-22 06:03:42.076 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779613641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:42.881 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257080686-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c302b98 at consistency null
2022-03-22 06:03:42.881 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257080686-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:42.883 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:03:47.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2021967222-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@587f0948 at consistency null
2022-03-22 06:03:47.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2021967222-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:47.948 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390949498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31e7c273 at consistency null
2022-03-22 06:03:47.948 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390949498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:48.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109386983-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@270e4bcb at consistency null
2022-03-22 06:03:48.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109386983-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:48.171 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472283284-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2aefc859 at consistency null
2022-03-22 06:03:48.171 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472283284-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:48.398 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1113043219-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@afaa105 at consistency null
2022-03-22 06:03:48.399 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1113043219-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:48.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1052056985-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5811e73 at consistency null
2022-03-22 06:03:48.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1052056985-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:49.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276384908-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50845d1b at consistency null
2022-03-22 06:03:49.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276384908-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:49.975 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1060402383-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fdfa52 at consistency null
2022-03-22 06:03:49.975 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1060402383-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:50.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1075090707-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b509141 at consistency null
2022-03-22 06:03:50.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1075090707-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:50.808 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:03:55.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1448808797-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@562f7ede at consistency null
2022-03-22 06:03:55.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1448808797-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:55.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [211561014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24318d85 at consistency null
2022-03-22 06:03:55.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [211561014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:56.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269657446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f75dc3b at consistency null
2022-03-22 06:03:56.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269657446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:56.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1951008378-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10f0860d at consistency null
2022-03-22 06:03:56.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1951008378-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:56.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2048204887-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d84c343 at consistency null
2022-03-22 06:03:56.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2048204887-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:56.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397764793-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15f13cd3 at consistency null
2022-03-22 06:03:56.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397764793-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:57.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [311023292-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7404ae52 at consistency null
2022-03-22 06:03:57.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [311023292-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:57.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [919890753-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60d08e31 at consistency null
2022-03-22 06:03:57.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [919890753-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:58.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703998036-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b55c7b3 at consistency null
2022-03-22 06:03:58.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703998036-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:03:58.647 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:04:03.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499250431-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67de7bae at consistency null
2022-03-22 06:04:03.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499250431-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:03.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96251742-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36f92f39 at consistency null
2022-03-22 06:04:03.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96251742-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:03.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [271008459-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52bc9ed1 at consistency null
2022-03-22 06:04:03.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [271008459-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:04.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1917259977-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@701a36a at consistency null
2022-03-22 06:04:04.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1917259977-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:04.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1322316600-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1316bb2f at consistency null
2022-03-22 06:04:04.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1322316600-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:04.553 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [787899363-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ce9ce93 at consistency null
2022-03-22 06:04:04.553 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [787899363-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:04.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82664940-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a158fc9 at consistency null
2022-03-22 06:04:04.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82664940-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:05.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315362016-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50856d12 at consistency null
2022-03-22 06:04:05.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315362016-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:05.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [132560679-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@766c7bee at consistency null
2022-03-22 06:04:05.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [132560679-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:05.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [437396803-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5453aee6 at consistency null
2022-03-22 06:04:05.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [437396803-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:06.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1962809808-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22ce0862 at consistency null
2022-03-22 06:04:06.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1962809808-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:07.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214372987-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59d80cd4 at consistency null
2022-03-22 06:04:07.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214372987-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:07.021 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:04:11.869 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:04:11.900 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:04:12.041 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051378507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cfebf0a at consistency null
2022-03-22 06:04:12.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051378507-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:12.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890880480-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5131ced at consistency null
2022-03-22 06:04:12.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890880480-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:12.197 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825064970-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@574c8935 at consistency null
2022-03-22 06:04:12.197 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825064970-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:12.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [88995486-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e6b81a4 at consistency null
2022-03-22 06:04:12.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [88995486-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:12.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1838534119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7754618c at consistency null
2022-03-22 06:04:12.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1838534119-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:12.566 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [385353588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a3defff at consistency null
2022-03-22 06:04:12.566 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [385353588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:12.787 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1291839119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@598bd6ac at consistency null
2022-03-22 06:04:12.788 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1291839119-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:13.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [653004936-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bff1a98 at consistency null
2022-03-22 06:04:13.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [653004936-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:13.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1700072263-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f734a6a at consistency null
2022-03-22 06:04:13.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1700072263-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:13.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [868250430-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a340c6c at consistency null
2022-03-22 06:04:13.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [868250430-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:14.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12070999-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d0d9730 at consistency null
2022-03-22 06:04:14.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12070999-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:15.569 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175849112-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e9575d3 at consistency null
2022-03-22 06:04:15.570 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175849112-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:15.571 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:04:20.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [190065710-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bb04a71 at consistency null
2022-03-22 06:04:20.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [190065710-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:20.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1002301433-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11dd7407 at consistency null
2022-03-22 06:04:20.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1002301433-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:20.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2051428945-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6098d51b at consistency null
2022-03-22 06:04:20.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2051428945-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:20.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [207804233-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65261f36 at consistency null
2022-03-22 06:04:20.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [207804233-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:21.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [645771191-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c6c3221 at consistency null
2022-03-22 06:04:21.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [645771191-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:21.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1915426478-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36eb1ae7 at consistency null
2022-03-22 06:04:21.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1915426478-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:21.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [976688736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10fd212c at consistency null
2022-03-22 06:04:21.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [976688736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:21.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [544556784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@457753aa at consistency null
2022-03-22 06:04:21.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [544556784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:22.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1799247198-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68279c1f at consistency null
2022-03-22 06:04:22.043 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1799247198-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:22.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099604209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6759fde at consistency null
2022-03-22 06:04:22.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099604209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:22.895 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1124874934-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f783a7d at consistency null
2022-03-22 06:04:22.895 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1124874934-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:23.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1237628641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ddc3b at consistency null
2022-03-22 06:04:23.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1237628641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:24.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355390498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@547e6f4d at consistency null
2022-03-22 06:04:24.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355390498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:24.156 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:04:29.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1146106338-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7edf8173 at consistency null
2022-03-22 06:04:29.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1146106338-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:29.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1722509908-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40caf707 at consistency null
2022-03-22 06:04:29.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1722509908-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:29.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171224030-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ea86721 at consistency null
2022-03-22 06:04:29.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171224030-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:29.653 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [370150452-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@90b61be at consistency null
2022-03-22 06:04:29.654 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [370150452-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:30.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1393221685-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b5dbdf5 at consistency null
2022-03-22 06:04:30.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1393221685-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:30.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1859049930-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cb80156 at consistency null
2022-03-22 06:04:30.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1859049930-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:32.673 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099687318-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7144ec8f at consistency null
2022-03-22 06:04:32.673 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099687318-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:32.701 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:04:37.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475096639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23906588 at consistency null
2022-03-22 06:04:37.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1475096639-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:37.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407416767-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45c4c61a at consistency null
2022-03-22 06:04:37.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1407416767-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:37.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1274804864-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@192add45 at consistency null
2022-03-22 06:04:37.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1274804864-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:38.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1430389052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fdc7a7b at consistency null
2022-03-22 06:04:38.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1430389052-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:38.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662627427-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fc0ea2d at consistency null
2022-03-22 06:04:38.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662627427-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:38.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [729746750-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cd715d9 at consistency null
2022-03-22 06:04:38.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [729746750-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:39.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1069866283-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43d821f3 at consistency null
2022-03-22 06:04:39.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1069866283-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:41.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2060083214-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f7b9451 at consistency null
2022-03-22 06:04:41.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2060083214-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:41.428 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:04:41.901 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:04:41.931 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:04:46.486 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1410245506-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7028a11 at consistency null
2022-03-22 06:04:46.486 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1410245506-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:46.549 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [130761408-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33cf3ef5 at consistency null
2022-03-22 06:04:46.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [130761408-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:46.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [511703703-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@311a56c6 at consistency null
2022-03-22 06:04:46.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [511703703-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:46.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [924590918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27095a8 at consistency null
2022-03-22 06:04:46.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [924590918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:46.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372216980-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11b1d5f5 at consistency null
2022-03-22 06:04:46.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372216980-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:47.047 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261378003-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bc379a0 at consistency null
2022-03-22 06:04:47.047 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261378003-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:47.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [711326051-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@388cb878 at consistency null
2022-03-22 06:04:47.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [711326051-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:47.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2083912058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e57a714 at consistency null
2022-03-22 06:04:47.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2083912058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:47.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [374626652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21d6d356 at consistency null
2022-03-22 06:04:47.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [374626652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:48.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629554116-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6198a0b5 at consistency null
2022-03-22 06:04:48.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629554116-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:49.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [548237558-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c0a9b86 at consistency null
2022-03-22 06:04:49.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [548237558-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:49.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1929425840-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28bddccd at consistency null
2022-03-22 06:04:49.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1929425840-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:49.834 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:04:54.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1193970229-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c844c67 at consistency null
2022-03-22 06:04:54.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1193970229-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:55.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399243023-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64ba20da at consistency null
2022-03-22 06:04:55.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399243023-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:55.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [411509077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c742e1d at consistency null
2022-03-22 06:04:55.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [411509077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:55.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1861000390-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ae9dcea at consistency null
2022-03-22 06:04:55.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1861000390-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:55.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2105853964-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a04aa4 at consistency null
2022-03-22 06:04:55.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2105853964-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:55.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [412028938-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31a9161a at consistency null
2022-03-22 06:04:55.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [412028938-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:55.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [966989767-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ce0d2f4 at consistency null
2022-03-22 06:04:55.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [966989767-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:56.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866363033-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@188a5f0e at consistency null
2022-03-22 06:04:56.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866363033-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:56.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1748819017-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ef9c3d1 at consistency null
2022-03-22 06:04:56.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1748819017-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:56.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139693572-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67990fd2 at consistency null
2022-03-22 06:04:56.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139693572-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:57.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703573478-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d9416a7 at consistency null
2022-03-22 06:04:57.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703573478-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:04:57.643 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:05:02.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [131485996-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57ac5834 at consistency null
2022-03-22 06:05:02.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [131485996-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:02.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [151390987-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71bee5b at consistency null
2022-03-22 06:05:02.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [151390987-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:02.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [489789313-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28d386f5 at consistency null
2022-03-22 06:05:02.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [489789313-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:03.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1239783772-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15cf9c66 at consistency null
2022-03-22 06:05:03.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1239783772-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:03.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2089807485-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20171b13 at consistency null
2022-03-22 06:05:03.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2089807485-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:03.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1164508340-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7826407c at consistency null
2022-03-22 06:05:03.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1164508340-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:04.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1425335815-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cf685a3 at consistency null
2022-03-22 06:05:04.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1425335815-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:05.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [826194620-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25330cf7 at consistency null
2022-03-22 06:05:05.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [826194620-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:05.855 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [843142880-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d6454c1 at consistency null
2022-03-22 06:05:05.855 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [843142880-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:05.857 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:05:06.759 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 06:05:06.768 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-46, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 06:05:06.768 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-46, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 06:05:06.768 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-46, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 06:05:06.770 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-46, inFlight=0, closed=true] closing connection
2022-03-22 06:05:06.772 DEBUG 10 --- [uster-worker-83] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 06:05:06.778 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 06:05:06.778 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 600000 ms
2022-03-22 06:05:10.868 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [818839872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@770bd6b1 at consistency null
2022-03-22 06:05:10.868 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [818839872-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:10.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1140432256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f289e6e at consistency null
2022-03-22 06:05:10.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1140432256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:11.050 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123089945-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32416fc8 at consistency null
2022-03-22 06:05:11.050 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [123089945-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:11.257 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1066959941-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d9ce71d at consistency null
2022-03-22 06:05:11.257 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1066959941-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:11.666 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2065225335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40b1241 at consistency null
2022-03-22 06:05:11.666 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2065225335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:11.932 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:05:11.939 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:05:12.072 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1946304777-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f1b3740 at consistency null
2022-03-22 06:05:12.073 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1946304777-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:12.891 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1120032108-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@769d2e44 at consistency null
2022-03-22 06:05:12.891 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1120032108-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:13.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [956521437-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c739b85 at consistency null
2022-03-22 06:05:13.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [956521437-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:14.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [76909028-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60360f50 at consistency null
2022-03-22 06:05:14.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [76909028-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:14.535 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:05:19.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1652136566-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76019521 at consistency null
2022-03-22 06:05:19.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1652136566-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:19.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851495520-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69f16aab at consistency null
2022-03-22 06:05:19.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851495520-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:19.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [297710166-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@497e6ba3 at consistency null
2022-03-22 06:05:19.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [297710166-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:20.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961332012-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19e2b153 at consistency null
2022-03-22 06:05:20.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961332012-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:20.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390612057-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63b087d3 at consistency null
2022-03-22 06:05:20.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390612057-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:21.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [594480922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1760d3c at consistency null
2022-03-22 06:05:21.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [594480922-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:22.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202136355-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b6f4dc2 at consistency null
2022-03-22 06:05:22.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202136355-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:22.232 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:05:27.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [414042968-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@370b1bd5 at consistency null
2022-03-22 06:05:27.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [414042968-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:27.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1293041304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50e29cf8 at consistency null
2022-03-22 06:05:27.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1293041304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:27.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [375324671-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61cc7435 at consistency null
2022-03-22 06:05:27.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [375324671-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:27.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1747865130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@200c3493 at consistency null
2022-03-22 06:05:27.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1747865130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:27.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [506441596-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f3656f4 at consistency null
2022-03-22 06:05:27.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [506441596-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:28.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1108718815-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bf716b6 at consistency null
2022-03-22 06:05:28.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1108718815-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:28.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [592775781-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@918a621 at consistency null
2022-03-22 06:05:28.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [592775781-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:28.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [965758779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48f1b33d at consistency null
2022-03-22 06:05:28.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [965758779-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:28.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1520911162-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@350ddd21 at consistency null
2022-03-22 06:05:28.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1520911162-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:28.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897296381-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fc2f90 at consistency null
2022-03-22 06:05:28.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897296381-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:29.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1056347413-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a99a8c0 at consistency null
2022-03-22 06:05:29.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1056347413-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:29.807 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:05:34.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [86305895-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fb492f5 at consistency null
2022-03-22 06:05:34.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [86305895-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:34.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1702543722-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@548ba2f7 at consistency null
2022-03-22 06:05:34.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1702543722-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:34.972 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [223794807-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33ec9759 at consistency null
2022-03-22 06:05:34.973 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [223794807-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:35.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [126002421-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34b4e64c at consistency null
2022-03-22 06:05:35.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [126002421-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:35.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [635910070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f3ce915 at consistency null
2022-03-22 06:05:35.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [635910070-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:35.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [379966449-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16296c1f at consistency null
2022-03-22 06:05:35.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [379966449-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:35.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [763058748-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@412877dc at consistency null
2022-03-22 06:05:35.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [763058748-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:36.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070397034-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52bd6f33 at consistency null
2022-03-22 06:05:36.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070397034-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:37.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596257954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a314030 at consistency null
2022-03-22 06:05:37.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596257954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:38.764 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461916537-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77a598fd at consistency null
2022-03-22 06:05:38.764 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461916537-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:38.767 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:05:40.530 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 06:05:40.539 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-70, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 06:05:40.539 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-70, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 06:05:40.539 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-70, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 06:05:40.541 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-70, inFlight=0, closed=true] closing connection
2022-03-22 06:05:40.542 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 06:05:40.543 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 600000 ms
2022-03-22 06:05:40.544 DEBUG 10 --- [uster-worker-84] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 06:05:41.961 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:05:42.011 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:05:43.779 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1825921678-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75b18742 at consistency null
2022-03-22 06:05:43.779 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1825921678-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:43.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [888315608-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c4cba30 at consistency null
2022-03-22 06:05:43.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [888315608-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:43.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181703128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cdc4349 at consistency null
2022-03-22 06:05:43.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1181703128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:44.187 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536150509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37654c5f at consistency null
2022-03-22 06:05:44.187 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536150509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:44.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504852407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e175666 at consistency null
2022-03-22 06:05:44.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504852407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:44.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2039904263-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67a3e8ae at consistency null
2022-03-22 06:05:44.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2039904263-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:44.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1167805814-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@696b3918 at consistency null
2022-03-22 06:05:44.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1167805814-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:45.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1918293569-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@747197de at consistency null
2022-03-22 06:05:45.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1918293569-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:46.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [979445309-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a14b50 at consistency null
2022-03-22 06:05:46.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [979445309-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:47.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887254326-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78bf2572 at consistency null
2022-03-22 06:05:47.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887254326-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:47.708 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:05:52.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1863464114-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48ac7084 at consistency null
2022-03-22 06:05:52.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1863464114-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:52.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724702159-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a6ae7b0 at consistency null
2022-03-22 06:05:52.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724702159-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:53.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407996506-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b7fc13a at consistency null
2022-03-22 06:05:53.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407996506-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:53.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [17615073-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10fcae81 at consistency null
2022-03-22 06:05:53.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [17615073-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:53.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2143669206-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e803f37 at consistency null
2022-03-22 06:05:53.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2143669206-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:53.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1154618250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13ba0030 at consistency null
2022-03-22 06:05:53.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1154618250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:53.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1347768338-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5689656c at consistency null
2022-03-22 06:05:53.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1347768338-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:54.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [939114809-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23f4419a at consistency null
2022-03-22 06:05:54.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [939114809-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:55.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1723017067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cf2ab54 at consistency null
2022-03-22 06:05:55.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1723017067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:56.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [841561716-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f8a8b7a at consistency null
2022-03-22 06:05:56.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [841561716-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:05:56.033 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:06:01.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1216751075-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2352fe2 at consistency null
2022-03-22 06:06:01.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1216751075-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:01.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [831509955-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cc46af at consistency null
2022-03-22 06:06:01.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [831509955-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:01.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1316500710-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a2de589 at consistency null
2022-03-22 06:06:01.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1316500710-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:01.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899228107-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b766ed at consistency null
2022-03-22 06:06:01.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899228107-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:01.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1279894438-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a561306 at consistency null
2022-03-22 06:06:01.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1279894438-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:01.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [61298209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bcec1dd at consistency null
2022-03-22 06:06:01.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [61298209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:01.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [209161655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6eea7480 at consistency null
2022-03-22 06:06:01.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [209161655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:01.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1889297035-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54d0e7be at consistency null
2022-03-22 06:06:01.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1889297035-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:01.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1900653485-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e040001 at consistency null
2022-03-22 06:06:01.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1900653485-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:02.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1214709243-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7379e035 at consistency null
2022-03-22 06:06:02.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1214709243-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:02.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026687349-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e8ac4ae at consistency null
2022-03-22 06:06:02.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026687349-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:02.745 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809282781-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1db437d6 at consistency null
2022-03-22 06:06:02.745 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809282781-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:03.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1467989193-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73de2869 at consistency null
2022-03-22 06:06:03.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1467989193-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:04.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671685368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f9a066f at consistency null
2022-03-22 06:06:04.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671685368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:04.456 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:06:09.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914422767-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@572d0486 at consistency null
2022-03-22 06:06:09.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914422767-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:09.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1722880636-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b1b9332 at consistency null
2022-03-22 06:06:09.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1722880636-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:09.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1283613591-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69847358 at consistency null
2022-03-22 06:06:09.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1283613591-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:09.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [750546469-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7632b979 at consistency null
2022-03-22 06:06:09.729 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [750546469-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:09.946 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [491956190-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49849a0d at consistency null
2022-03-22 06:06:09.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [491956190-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:10.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931375866-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e31120c at consistency null
2022-03-22 06:06:10.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931375866-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:10.759 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752828373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34e5b021 at consistency null
2022-03-22 06:06:10.759 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752828373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:11.563 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667084589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@577f2eb5 at consistency null
2022-03-22 06:06:11.563 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667084589-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:12.012 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:06:12.018 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:06:13.168 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1807320783-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a4692a5 at consistency null
2022-03-22 06:06:13.168 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1807320783-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:13.170 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:06:18.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1803585254-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c115693 at consistency null
2022-03-22 06:06:18.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1803585254-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:18.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1047805396-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55ec8388 at consistency null
2022-03-22 06:06:18.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1047805396-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:18.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808331246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bc81506 at consistency null
2022-03-22 06:06:18.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808331246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:19.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394678490-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75956360 at consistency null
2022-03-22 06:06:19.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394678490-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:19.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2019447674-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33f6068f at consistency null
2022-03-22 06:06:19.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2019447674-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:19.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742428984-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b723ba at consistency null
2022-03-22 06:06:19.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742428984-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:19.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [646648186-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@396c66cf at consistency null
2022-03-22 06:06:19.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [646648186-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:19.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [61773297-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@673c9ce1 at consistency null
2022-03-22 06:06:19.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [61773297-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:20.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165794685-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64fd2cf7 at consistency null
2022-03-22 06:06:20.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165794685-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:20.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [878599138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fec7921 at consistency null
2022-03-22 06:06:20.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [878599138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:20.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243651282-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56c260d at consistency null
2022-03-22 06:06:20.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243651282-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:21.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1029081466-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63469d5 at consistency null
2022-03-22 06:06:21.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1029081466-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:21.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [984338326-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b237770 at consistency null
2022-03-22 06:06:21.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [984338326-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:22.157 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095845994-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37ba2244 at consistency null
2022-03-22 06:06:22.157 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095845994-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:22.158 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:06:27.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [845469481-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@538226f0 at consistency null
2022-03-22 06:06:27.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [845469481-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:27.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005622598-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76dd8794 at consistency null
2022-03-22 06:06:27.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005622598-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:27.394 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1606626001-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6655f6c8 at consistency null
2022-03-22 06:06:27.395 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1606626001-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:27.558 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134566401-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9b4044 at consistency null
2022-03-22 06:06:27.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1134566401-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:27.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1466521283-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42ec4301 at consistency null
2022-03-22 06:06:27.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1466521283-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:28.249 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1745658829-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58178def at consistency null
2022-03-22 06:06:28.249 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1745658829-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:29.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [786405823-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11bdd393 at consistency null
2022-03-22 06:06:29.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [786405823-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:30.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1843185777-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b7e2880 at consistency null
2022-03-22 06:06:30.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1843185777-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:30.126 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:06:35.134 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1881035847-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6baa9ea5 at consistency null
2022-03-22 06:06:35.134 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1881035847-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:35.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2097251138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f361eca at consistency null
2022-03-22 06:06:35.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2097251138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:35.245 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [801242940-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70e865f4 at consistency null
2022-03-22 06:06:35.245 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [801242940-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:35.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617952757-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a687941 at consistency null
2022-03-22 06:06:35.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617952757-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:35.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [652992836-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7508891f at consistency null
2022-03-22 06:06:35.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [652992836-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:35.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1468104008-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4af7193a at consistency null
2022-03-22 06:06:35.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1468104008-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:36.162 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1998143643-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79030255 at consistency null
2022-03-22 06:06:36.162 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1998143643-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:37.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2062461200-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7028ce45 at consistency null
2022-03-22 06:06:37.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2062461200-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:38.631 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [66492104-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41d9f12e at consistency null
2022-03-22 06:06:38.631 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [66492104-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:38.632 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:06:42.043 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:06:42.063 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:06:43.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1598144707-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33951ee6 at consistency null
2022-03-22 06:06:43.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1598144707-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:43.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1658936872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5265b36b at consistency null
2022-03-22 06:06:43.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1658936872-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:43.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1841880397-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64090584 at consistency null
2022-03-22 06:06:43.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1841880397-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:44.056 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1514100458-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a57d8aa at consistency null
2022-03-22 06:06:44.056 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1514100458-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:44.284 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [195674806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@712e9928 at consistency null
2022-03-22 06:06:44.284 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [195674806-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:44.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674698229-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@eaf828d at consistency null
2022-03-22 06:06:44.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674698229-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:44.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213748855-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22a683f9 at consistency null
2022-03-22 06:06:44.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213748855-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:45.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1752224144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76e09d8f at consistency null
2022-03-22 06:06:45.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1752224144-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:47.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [755565146-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7302bda2 at consistency null
2022-03-22 06:06:47.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [755565146-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:47.406 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:06:52.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079659643-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cbf2a5 at consistency null
2022-03-22 06:06:52.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079659643-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:52.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1245397481-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1764102e at consistency null
2022-03-22 06:06:52.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1245397481-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:52.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [615118191-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@659aeba8 at consistency null
2022-03-22 06:06:52.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [615118191-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:52.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1104619634-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@393a514 at consistency null
2022-03-22 06:06:52.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1104619634-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:53.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1400675751-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d133119 at consistency null
2022-03-22 06:06:53.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1400675751-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:53.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129884200-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a41bff5 at consistency null
2022-03-22 06:06:53.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129884200-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:53.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [682700193-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5834cebe at consistency null
2022-03-22 06:06:53.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [682700193-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:53.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348277373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46917ba9 at consistency null
2022-03-22 06:06:53.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348277373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:54.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736129140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ce119b at consistency null
2022-03-22 06:06:54.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736129140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:54.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1643435353-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6597a820 at consistency null
2022-03-22 06:06:54.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1643435353-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:54.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074344163-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b28db12 at consistency null
2022-03-22 06:06:54.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074344163-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:55.063 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026842723-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56a9b7ed at consistency null
2022-03-22 06:06:55.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026842723-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:55.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1188628372-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2419366a at consistency null
2022-03-22 06:06:55.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1188628372-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:56.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [29379595-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c175c61 at consistency null
2022-03-22 06:06:56.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [29379595-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:06:56.517 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:07:01.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147319544-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@213ae8ba at consistency null
2022-03-22 06:07:01.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147319544-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:01.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165303729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a3735af at consistency null
2022-03-22 06:07:01.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165303729-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:01.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129046201-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57f0eee5 at consistency null
2022-03-22 06:07:01.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129046201-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:02.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68334160-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11c2542e at consistency null
2022-03-22 06:07:02.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68334160-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:02.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859117568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67cb1194 at consistency null
2022-03-22 06:07:02.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859117568-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:02.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754903562-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6452ca8f at consistency null
2022-03-22 06:07:02.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754903562-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:02.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1737651070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34858726 at consistency null
2022-03-22 06:07:02.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1737651070-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:02.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1979964910-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37e5c6bc at consistency null
2022-03-22 06:07:02.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1979964910-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:02.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835871798-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@89d3665 at consistency null
2022-03-22 06:07:02.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835871798-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:03.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215430261-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a69834b at consistency null
2022-03-22 06:07:03.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215430261-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:03.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1007451526-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62cd9797 at consistency null
2022-03-22 06:07:03.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1007451526-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:03.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1409825604-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b047d54 at consistency null
2022-03-22 06:07:03.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1409825604-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:03.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1343267693-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1acb524f at consistency null
2022-03-22 06:07:03.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1343267693-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:04.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391474508-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c0ef986 at consistency null
2022-03-22 06:07:04.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391474508-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:04.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223344749-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22f1b2bf at consistency null
2022-03-22 06:07:04.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223344749-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:04.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571619375-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ac78568 at consistency null
2022-03-22 06:07:04.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571619375-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:05.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [150198588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3adee955 at consistency null
2022-03-22 06:07:05.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [150198588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:05.412 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:07:10.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879819363-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e0cd78c at consistency null
2022-03-22 06:07:10.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879819363-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:10.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [714583850-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@826e7e0 at consistency null
2022-03-22 06:07:10.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [714583850-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:10.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [530476077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45ff33f0 at consistency null
2022-03-22 06:07:10.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [530476077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:10.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [162692315-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bccabf3 at consistency null
2022-03-22 06:07:10.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [162692315-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:10.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [983706732-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33d4afb1 at consistency null
2022-03-22 06:07:10.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [983706732-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:11.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769342189-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cf247cc at consistency null
2022-03-22 06:07:11.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769342189-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:11.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1095292031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@302e8511 at consistency null
2022-03-22 06:07:11.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1095292031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:11.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1527135273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ec323d4 at consistency null
2022-03-22 06:07:11.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1527135273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:11.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1894558313-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@587ca49a at consistency null
2022-03-22 06:07:11.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1894558313-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:12.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [76868299-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46ae447e at consistency null
2022-03-22 06:07:12.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [76868299-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:12.073 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:07:12.108 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:07:12.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [66600202-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ec47b7 at consistency null
2022-03-22 06:07:12.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [66600202-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:13.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1777708584-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@556619bd at consistency null
2022-03-22 06:07:13.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1777708584-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:13.623 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:07:18.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095050397-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68833fff at consistency null
2022-03-22 06:07:18.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095050397-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:18.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1597935999-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c3a1b25 at consistency null
2022-03-22 06:07:18.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1597935999-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:18.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [502065151-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c8b2eea at consistency null
2022-03-22 06:07:18.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [502065151-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:19.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1496096388-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59456337 at consistency null
2022-03-22 06:07:19.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1496096388-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:19.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1510221882-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28c8c327 at consistency null
2022-03-22 06:07:19.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1510221882-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:19.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1897594732-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24271628 at consistency null
2022-03-22 06:07:19.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1897594732-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:20.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [187903407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@413e9038 at consistency null
2022-03-22 06:07:20.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [187903407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:20.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [911624947-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33900ff3 at consistency null
2022-03-22 06:07:20.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [911624947-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:21.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298719227-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d2f235d at consistency null
2022-03-22 06:07:21.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298719227-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:21.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806665381-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41a5e886 at consistency null
2022-03-22 06:07:21.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806665381-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:22.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2036568106-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@742f0fda at consistency null
2022-03-22 06:07:22.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2036568106-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:22.405 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:07:27.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103856167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4773793b at consistency null
2022-03-22 06:07:27.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103856167-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:27.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2005605705-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63629f43 at consistency null
2022-03-22 06:07:27.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2005605705-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:27.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1619408819-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@dedba71 at consistency null
2022-03-22 06:07:27.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1619408819-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:27.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83958089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b025c9 at consistency null
2022-03-22 06:07:27.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83958089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:27.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1246325434-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d562e65 at consistency null
2022-03-22 06:07:27.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1246325434-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:28.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703368687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a392a0b at consistency null
2022-03-22 06:07:28.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703368687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:28.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [888004419-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ae7230f at consistency null
2022-03-22 06:07:28.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [888004419-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:29.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [959140806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a79c9f7 at consistency null
2022-03-22 06:07:29.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [959140806-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:31.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [943121123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2630f3db at consistency null
2022-03-22 06:07:31.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [943121123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:31.210 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:07:36.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741808165-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45bafc64 at consistency null
2022-03-22 06:07:36.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741808165-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:36.257 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025261147-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36b65087 at consistency null
2022-03-22 06:07:36.257 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025261147-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:36.290 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1581973341-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@328401d7 at consistency null
2022-03-22 06:07:36.291 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1581973341-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:36.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1365520269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@130b9277 at consistency null
2022-03-22 06:07:36.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1365520269-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:36.447 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961294158-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c005ae at consistency null
2022-03-22 06:07:36.447 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961294158-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:36.552 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1711157466-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f65f79 at consistency null
2022-03-22 06:07:36.552 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1711157466-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:36.655 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1816628015-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17ee6ca9 at consistency null
2022-03-22 06:07:36.655 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1816628015-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:36.761 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472893805-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58b9cc09 at consistency null
2022-03-22 06:07:36.762 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472893805-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:36.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1153545209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59555fc5 at consistency null
2022-03-22 06:07:36.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1153545209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:37.189 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1353524059-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6eefce01 at consistency null
2022-03-22 06:07:37.189 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1353524059-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:37.400 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [6538386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a807c51 at consistency null
2022-03-22 06:07:37.400 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [6538386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:37.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1659179346-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ccf208c at consistency null
2022-03-22 06:07:37.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1659179346-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:38.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [737294476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f97efa7 at consistency null
2022-03-22 06:07:38.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [737294476-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:38.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310145362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f98bb92 at consistency null
2022-03-22 06:07:38.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310145362-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:38.830 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:07:42.109 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:07:42.112 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:07:43.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [212164827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76f332c4 at consistency null
2022-03-22 06:07:43.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [212164827-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:43.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1182834588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13dda7d6 at consistency null
2022-03-22 06:07:43.893 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1182834588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:43.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2081221016-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43e798dd at consistency null
2022-03-22 06:07:43.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2081221016-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:43.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [218886570-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7999ef18 at consistency null
2022-03-22 06:07:43.974 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [218886570-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:44.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [705690472-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@169387fd at consistency null
2022-03-22 06:07:44.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [705690472-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:44.060 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1756495306-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@688e61ee at consistency null
2022-03-22 06:07:44.061 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1756495306-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:44.164 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1944750445-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17503d52 at consistency null
2022-03-22 06:07:44.164 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1944750445-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:44.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [566145051-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2187e18f at consistency null
2022-03-22 06:07:44.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [566145051-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:44.592 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950342143-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b21362d at consistency null
2022-03-22 06:07:44.593 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950342143-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:44.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [319235871-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30bbb3b6 at consistency null
2022-03-22 06:07:44.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [319235871-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:45.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [872506867-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2617b25e at consistency null
2022-03-22 06:07:45.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [872506867-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:45.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626023168-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3802b48a at consistency null
2022-03-22 06:07:45.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1626023168-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:45.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1786899137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a307df9 at consistency null
2022-03-22 06:07:45.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1786899137-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:45.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083447121-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@777b4b0a at consistency null
2022-03-22 06:07:45.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083447121-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:46.243 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2131090509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ce592f9 at consistency null
2022-03-22 06:07:46.243 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2131090509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:46.655 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1191130504-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@416ecd0b at consistency null
2022-03-22 06:07:46.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1191130504-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:47.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [600639604-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58a8bf6a at consistency null
2022-03-22 06:07:47.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [600639604-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:47.509 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:07:52.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1132703171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75a69c6e at consistency null
2022-03-22 06:07:52.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1132703171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:52.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [305522250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3514aa79 at consistency null
2022-03-22 06:07:52.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [305522250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:52.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769250033-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a53f45 at consistency null
2022-03-22 06:07:52.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769250033-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:52.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721561494-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f6b9394 at consistency null
2022-03-22 06:07:52.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721561494-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:53.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [951039283-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5eef39fb at consistency null
2022-03-22 06:07:53.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [951039283-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:53.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [93651930-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e38073d at consistency null
2022-03-22 06:07:53.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [93651930-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:53.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1714952131-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65cc46a8 at consistency null
2022-03-22 06:07:53.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1714952131-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:53.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1730763475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26789eb6 at consistency null
2022-03-22 06:07:53.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1730763475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:54.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [577400104-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ff4b63 at consistency null
2022-03-22 06:07:54.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [577400104-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:55.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256647532-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f70a58e at consistency null
2022-03-22 06:07:55.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256647532-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:55.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703375689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@80e4bce at consistency null
2022-03-22 06:07:55.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703375689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:07:55.910 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:08:00.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [79597925-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ae1e957 at consistency null
2022-03-22 06:08:00.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [79597925-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:01.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1967960055-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11be133b at consistency null
2022-03-22 06:08:01.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1967960055-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:01.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [459646835-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1be88857 at consistency null
2022-03-22 06:08:01.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [459646835-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:01.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469047735-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61c2f0df at consistency null
2022-03-22 06:08:01.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469047735-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:01.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [667575706-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40c16e1 at consistency null
2022-03-22 06:08:01.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [667575706-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:02.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938034563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@654aea90 at consistency null
2022-03-22 06:08:02.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938034563-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:04.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [25240114-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a3f6c5e at consistency null
2022-03-22 06:08:04.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [25240114-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:04.244 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:08:09.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [691792878-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c8d27ce at consistency null
2022-03-22 06:08:09.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [691792878-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:09.300 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571405286-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28d29ba0 at consistency null
2022-03-22 06:08:09.300 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571405286-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:09.362 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [537996941-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a7eea40 at consistency null
2022-03-22 06:08:09.362 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [537996941-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:09.474 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680069135-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5be238cd at consistency null
2022-03-22 06:08:09.474 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680069135-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:09.586 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578889835-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ef4d728 at consistency null
2022-03-22 06:08:09.586 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578889835-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:09.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1313678019-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42475ccd at consistency null
2022-03-22 06:08:09.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1313678019-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:10.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736513729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f06922f at consistency null
2022-03-22 06:08:10.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1736513729-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:11.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [120106318-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d7d3540 at consistency null
2022-03-22 06:08:11.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [120106318-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:11.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [111251306-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45a4a716 at consistency null
2022-03-22 06:08:11.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [111251306-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:12.113 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:08:12.125 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:08:12.660 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [559803975-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e28d267 at consistency null
2022-03-22 06:08:12.660 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [559803975-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:12.661 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:08:17.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1316335037-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c78d883 at consistency null
2022-03-22 06:08:17.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1316335037-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:17.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [209005536-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@287b3579 at consistency null
2022-03-22 06:08:17.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [209005536-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:17.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [111493083-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4887ab77 at consistency null
2022-03-22 06:08:17.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [111493083-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:18.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1220859550-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5024e3ea at consistency null
2022-03-22 06:08:18.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1220859550-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:18.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [733975166-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a2285ad at consistency null
2022-03-22 06:08:18.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [733975166-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:18.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [246449858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b83a56d at consistency null
2022-03-22 06:08:18.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [246449858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:18.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [558115624-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6eaa6979 at consistency null
2022-03-22 06:08:18.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [558115624-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:18.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1339718072-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1966fcf5 at consistency null
2022-03-22 06:08:18.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1339718072-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:18.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [413216351-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67171d99 at consistency null
2022-03-22 06:08:18.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [413216351-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:19.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [304950559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61b974db at consistency null
2022-03-22 06:08:19.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [304950559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:19.940 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [305154329-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ce53834 at consistency null
2022-03-22 06:08:19.940 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [305154329-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:20.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [238810462-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@724af609 at consistency null
2022-03-22 06:08:20.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [238810462-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:20.808 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:08:25.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [528081168-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d6ec60c at consistency null
2022-03-22 06:08:25.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [528081168-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:25.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1690792501-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7db70d6 at consistency null
2022-03-22 06:08:25.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1690792501-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:26.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957664565-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@720adc81 at consistency null
2022-03-22 06:08:26.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957664565-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:26.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [628039163-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57559677 at consistency null
2022-03-22 06:08:26.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [628039163-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:26.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1976441127-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c507437 at consistency null
2022-03-22 06:08:26.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1976441127-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:26.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [709335921-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5051f8ab at consistency null
2022-03-22 06:08:26.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [709335921-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:26.650 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [584689459-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@749bd8f4 at consistency null
2022-03-22 06:08:26.650 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [584689459-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:26.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2139416150-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f060e3e at consistency null
2022-03-22 06:08:26.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2139416150-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:27.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2055659424-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f82225 at consistency null
2022-03-22 06:08:27.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2055659424-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:27.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821134738-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f330631 at consistency null
2022-03-22 06:08:27.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821134738-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:28.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [58957071-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7545395b at consistency null
2022-03-22 06:08:28.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [58957071-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:28.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1542286719-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a8dae33 at consistency null
2022-03-22 06:08:28.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1542286719-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:29.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391823997-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b25444d at consistency null
2022-03-22 06:08:29.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391823997-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:29.517 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:08:34.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [430243803-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c60c996 at consistency null
2022-03-22 06:08:34.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [430243803-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:34.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [105931269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1033aea0 at consistency null
2022-03-22 06:08:34.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [105931269-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:34.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921071909-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37e876a3 at consistency null
2022-03-22 06:08:34.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921071909-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:34.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012091566-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16ce4b3f at consistency null
2022-03-22 06:08:34.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012091566-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:34.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1225418671-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ca80c09 at consistency null
2022-03-22 06:08:34.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1225418671-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:35.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1288664257-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51e3ff92 at consistency null
2022-03-22 06:08:35.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1288664257-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:35.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235569444-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@254ad663 at consistency null
2022-03-22 06:08:35.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235569444-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:35.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1323920858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4139f646 at consistency null
2022-03-22 06:08:35.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1323920858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:35.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1413762231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d6d622c at consistency null
2022-03-22 06:08:35.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1413762231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:35.740 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662665602-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13bc197c at consistency null
2022-03-22 06:08:35.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662665602-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:35.969 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [8767662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60363ca6 at consistency null
2022-03-22 06:08:35.970 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [8767662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:36.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [84402905-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6eb36d20 at consistency null
2022-03-22 06:08:36.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [84402905-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:36.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1943021370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58f90ed8 at consistency null
2022-03-22 06:08:36.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1943021370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:37.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1122466961-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bfd9b42 at consistency null
2022-03-22 06:08:37.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1122466961-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:38.160 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820921051-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a7f5197 at consistency null
2022-03-22 06:08:38.161 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820921051-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:38.168 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:08:42.130 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:08:42.135 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:08:43.182 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1445079737-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fd0f995 at consistency null
2022-03-22 06:08:43.182 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1445079737-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:43.239 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607731376-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f4fa979 at consistency null
2022-03-22 06:08:43.239 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607731376-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:43.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564728658-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@582f1659 at consistency null
2022-03-22 06:08:43.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564728658-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:43.365 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2030632437-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58dc1722 at consistency null
2022-03-22 06:08:43.366 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2030632437-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:43.472 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1043883919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50c841fd at consistency null
2022-03-22 06:08:43.472 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1043883919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:43.583 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1402540614-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@322e3efc at consistency null
2022-03-22 06:08:43.583 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1402540614-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:43.697 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284425258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12137b3e at consistency null
2022-03-22 06:08:43.697 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284425258-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:43.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [774528148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a4e0b59 at consistency null
2022-03-22 06:08:43.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [774528148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:44.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406358209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@381b7044 at consistency null
2022-03-22 06:08:44.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1406358209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:44.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1212379615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b817065 at consistency null
2022-03-22 06:08:44.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1212379615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:45.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197397585-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72423ea0 at consistency null
2022-03-22 06:08:45.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197397585-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:46.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [996998354-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@288ce772 at consistency null
2022-03-22 06:08:46.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [996998354-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:46.354 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:08:51.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1927341092-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bf63b77 at consistency null
2022-03-22 06:08:51.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1927341092-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:51.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [570342958-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b846834 at consistency null
2022-03-22 06:08:51.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [570342958-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:51.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270731294-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5850e6da at consistency null
2022-03-22 06:08:51.630 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270731294-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:51.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1291185316-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17282dd5 at consistency null
2022-03-22 06:08:51.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1291185316-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:51.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969007322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6dff91c4 at consistency null
2022-03-22 06:08:51.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969007322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:52.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1830434306-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75229187 at consistency null
2022-03-22 06:08:52.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1830434306-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:52.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1701945415-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a9bcfcf at consistency null
2022-03-22 06:08:52.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1701945415-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:52.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [100855937-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c0b19cc at consistency null
2022-03-22 06:08:52.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [100855937-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:52.674 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [812849067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6debd638 at consistency null
2022-03-22 06:08:52.675 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [812849067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:52.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [207986049-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c24ccab at consistency null
2022-03-22 06:08:52.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [207986049-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:53.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1694394322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78b58f57 at consistency null
2022-03-22 06:08:53.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1694394322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:53.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1978209588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c542cf4 at consistency null
2022-03-22 06:08:53.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1978209588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:54.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1569688173-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fbd7b50 at consistency null
2022-03-22 06:08:54.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1569688173-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:55.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2108071368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4556a398 at consistency null
2022-03-22 06:08:55.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2108071368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:08:55.006 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:09:00.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794225497-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c83ee13 at consistency null
2022-03-22 06:09:00.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794225497-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:00.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1460424070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68606f28 at consistency null
2022-03-22 06:09:00.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1460424070-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:00.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1995532776-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@185a4bdf at consistency null
2022-03-22 06:09:00.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1995532776-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:00.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1064362531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63764c55 at consistency null
2022-03-22 06:09:00.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1064362531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:00.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1489814505-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1076d55e at consistency null
2022-03-22 06:09:00.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1489814505-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:01.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1830405387-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b04e157 at consistency null
2022-03-22 06:09:01.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1830405387-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:01.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [282746729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31ca595b at consistency null
2022-03-22 06:09:01.233 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [282746729-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:01.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671336107-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@608fbfd3 at consistency null
2022-03-22 06:09:01.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671336107-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:01.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [376343497-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46f55ab9 at consistency null
2022-03-22 06:09:01.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [376343497-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:02.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [905717138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1aa70569 at consistency null
2022-03-22 06:09:02.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [905717138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:02.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1998521683-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@794487e7 at consistency null
2022-03-22 06:09:02.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1998521683-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:03.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1100981125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8647085 at consistency null
2022-03-22 06:09:03.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1100981125-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:03.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [600656060-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2acaa0ad at consistency null
2022-03-22 06:09:03.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [600656060-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:03.424 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:09:08.438 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2001287650-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@776c71ac at consistency null
2022-03-22 06:09:08.438 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2001287650-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:08.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029227973-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5597b06c at consistency null
2022-03-22 06:09:08.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2029227973-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:08.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1145866453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7edc0203 at consistency null
2022-03-22 06:09:08.625 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1145866453-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:08.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [218996808-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ec1abd at consistency null
2022-03-22 06:09:08.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [218996808-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:09.257 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859119138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1519ea3d at consistency null
2022-03-22 06:09:09.257 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859119138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:10.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1465659942-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7de52c22 at consistency null
2022-03-22 06:09:10.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1465659942-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:11.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2004222520-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6337e587 at consistency null
2022-03-22 06:09:11.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2004222520-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:11.696 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:09:12.169 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:09:12.185 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:09:16.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [395965310-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58fd68dc at consistency null
2022-03-22 06:09:16.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [395965310-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:16.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214317985-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@caddbdd at consistency null
2022-03-22 06:09:16.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214317985-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:16.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [977134877-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50ea0c12 at consistency null
2022-03-22 06:09:16.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [977134877-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:16.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2099074976-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bebaee3 at consistency null
2022-03-22 06:09:16.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2099074976-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:17.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633817470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18e398ca at consistency null
2022-03-22 06:09:17.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633817470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:17.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [409036116-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a72cb38 at consistency null
2022-03-22 06:09:17.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [409036116-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:17.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294036552-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a78d2f8 at consistency null
2022-03-22 06:09:17.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1294036552-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:17.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1519812635-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c92e7e2 at consistency null
2022-03-22 06:09:17.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1519812635-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:17.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1158000270-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8331b72 at consistency null
2022-03-22 06:09:17.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1158000270-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:17.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1457338559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64f43a47 at consistency null
2022-03-22 06:09:17.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1457338559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:18.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [273174274-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5827db33 at consistency null
2022-03-22 06:09:18.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [273174274-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:19.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2125501256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43919a22 at consistency null
2022-03-22 06:09:19.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2125501256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:20.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371794894-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53e9e1a9 at consistency null
2022-03-22 06:09:20.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371794894-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:20.639 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:09:25.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [697235828-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@168acf91 at consistency null
2022-03-22 06:09:25.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [697235828-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:25.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2070709016-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70927030 at consistency null
2022-03-22 06:09:25.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2070709016-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:26.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [20739566-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f9da48a at consistency null
2022-03-22 06:09:26.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [20739566-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:26.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1582567764-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2365fe79 at consistency null
2022-03-22 06:09:26.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1582567764-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:26.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [13567667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63d35bc3 at consistency null
2022-03-22 06:09:26.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [13567667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:26.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [530850289-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a3a479 at consistency null
2022-03-22 06:09:26.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [530850289-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:27.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2111153390-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5248f913 at consistency null
2022-03-22 06:09:27.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2111153390-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:27.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [66438321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e524315 at consistency null
2022-03-22 06:09:27.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [66438321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:28.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079830242-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2726065f at consistency null
2022-03-22 06:09:28.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079830242-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:28.723 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:09:33.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172487620-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a3beeae at consistency null
2022-03-22 06:09:33.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172487620-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:33.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [364938050-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f2e189b at consistency null
2022-03-22 06:09:33.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [364938050-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:33.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [943529223-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63a8bc80 at consistency null
2022-03-22 06:09:33.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [943529223-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:34.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123090498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74ab4a54 at consistency null
2022-03-22 06:09:34.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123090498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:34.277 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [777471178-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@118084f1 at consistency null
2022-03-22 06:09:34.278 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [777471178-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:34.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067190481-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e35bc9a at consistency null
2022-03-22 06:09:34.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067190481-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:34.972 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1872363264-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68df5560 at consistency null
2022-03-22 06:09:34.972 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1872363264-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:35.379 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340137835-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46884bc5 at consistency null
2022-03-22 06:09:35.379 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340137835-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:36.191 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1527090505-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53baf8e9 at consistency null
2022-03-22 06:09:36.191 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1527090505-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:36.192 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:09:41.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [67161208-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30084f68 at consistency null
2022-03-22 06:09:41.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [67161208-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:41.260 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [608260615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22ee685c at consistency null
2022-03-22 06:09:41.260 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [608260615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:41.369 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1826751581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26bcc9a7 at consistency null
2022-03-22 06:09:41.369 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1826751581-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:41.590 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461022029-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22450029 at consistency null
2022-03-22 06:09:41.590 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461022029-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:41.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421257046-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50446806 at consistency null
2022-03-22 06:09:41.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421257046-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:42.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [937719576-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13956da at consistency null
2022-03-22 06:09:42.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [937719576-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:42.197 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:09:42.200 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:09:42.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1264078165-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@410856ab at consistency null
2022-03-22 06:09:42.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1264078165-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:43.244 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [91684792-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@383117cd at consistency null
2022-03-22 06:09:43.244 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [91684792-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:44.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1120952623-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73750997 at consistency null
2022-03-22 06:09:44.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1120952623-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:44.868 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1444451034-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@704bb45c at consistency null
2022-03-22 06:09:44.868 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1444451034-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:44.869 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:09:49.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596149673-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9e361a4 at consistency null
2022-03-22 06:09:49.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596149673-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:50.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1838870730-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@395c802a at consistency null
2022-03-22 06:09:50.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1838870730-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:50.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [738344271-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62be1f27 at consistency null
2022-03-22 06:09:50.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [738344271-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:50.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1042216023-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ffaae78 at consistency null
2022-03-22 06:09:50.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1042216023-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:50.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063972922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d861862 at consistency null
2022-03-22 06:09:50.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2063972922-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:50.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1101207856-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@869e28 at consistency null
2022-03-22 06:09:50.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1101207856-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:50.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088156174-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1253fe38 at consistency null
2022-03-22 06:09:50.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088156174-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:51.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897794971-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8c00b64 at consistency null
2022-03-22 06:09:51.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897794971-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:51.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1669462907-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65e85716 at consistency null
2022-03-22 06:09:51.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1669462907-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:52.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [984749218-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51abe25d at consistency null
2022-03-22 06:09:52.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [984749218-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:52.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1944934710-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c04646e at consistency null
2022-03-22 06:09:52.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1944934710-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:53.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549620043-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5417cfa1 at consistency null
2022-03-22 06:09:53.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549620043-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:53.841 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:09:58.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [820715928-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73e072d2 at consistency null
2022-03-22 06:09:58.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [820715928-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:59.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [450241571-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dd7dff9 at consistency null
2022-03-22 06:09:59.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [450241571-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:59.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [65796774-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20fe339 at consistency null
2022-03-22 06:09:59.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [65796774-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:59.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899294875-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@688f5e1e at consistency null
2022-03-22 06:09:59.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [899294875-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:59.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553213009-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1809c73f at consistency null
2022-03-22 06:09:59.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553213009-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:59.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2001270492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5be2ebd0 at consistency null
2022-03-22 06:09:59.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2001270492-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:09:59.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732568665-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2609ddbe at consistency null
2022-03-22 06:09:59.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732568665-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:00.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340098889-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@346418cf at consistency null
2022-03-22 06:10:00.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340098889-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:00.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [842750026-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e72daa8 at consistency null
2022-03-22 06:10:00.501 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [842750026-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:00.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [41264839-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@467e16f5 at consistency null
2022-03-22 06:10:00.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [41264839-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:01.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2034645508-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5248cc95 at consistency null
2022-03-22 06:10:01.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2034645508-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:01.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [874618468-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20b25786 at consistency null
2022-03-22 06:10:01.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [874618468-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:02.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555069346-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18ad6ea2 at consistency null
2022-03-22 06:10:02.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555069346-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:02.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1935458157-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e520afc at consistency null
2022-03-22 06:10:02.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1935458157-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:02.724 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:10:07.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998391064-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a6a57c7 at consistency null
2022-03-22 06:10:07.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998391064-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:07.794 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [65558194-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8d6296b at consistency null
2022-03-22 06:10:07.794 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [65558194-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:07.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1556763140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bfd8ead at consistency null
2022-03-22 06:10:07.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1556763140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:08.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1257978151-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45c67ec4 at consistency null
2022-03-22 06:10:08.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1257978151-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:08.255 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1622910877-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2dfaddd4 at consistency null
2022-03-22 06:10:08.255 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1622910877-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:08.659 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [556216574-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14de3471 at consistency null
2022-03-22 06:10:08.659 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [556216574-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:09.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804457644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34148a1 at consistency null
2022-03-22 06:10:09.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804457644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:09.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1012021876-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75873653 at consistency null
2022-03-22 06:10:09.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1012021876-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:09.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23651628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e68514a at consistency null
2022-03-22 06:10:09.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23651628-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:10.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [424707391-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b70814a at consistency null
2022-03-22 06:10:10.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [424707391-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:10.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779185365-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62fdd8c5 at consistency null
2022-03-22 06:10:10.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779185365-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:11.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64844533-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e844f67 at consistency null
2022-03-22 06:10:11.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64844533-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:11.158 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:10:12.224 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:10:12.246 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:10:16.164 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1171961383-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62aaa9fc at consistency null
2022-03-22 06:10:16.165 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1171961383-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:16.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1488894686-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@dbf1e32 at consistency null
2022-03-22 06:10:16.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1488894686-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:16.278 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2097963736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@108c4b0 at consistency null
2022-03-22 06:10:16.278 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2097963736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:16.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [363785558-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@dfb4c81 at consistency null
2022-03-22 06:10:16.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [363785558-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:16.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [89459274-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3aa9e18b at consistency null
2022-03-22 06:10:16.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [89459274-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:16.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680540276-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f923662 at consistency null
2022-03-22 06:10:16.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680540276-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:16.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907314460-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8f84664 at consistency null
2022-03-22 06:10:16.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907314460-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:17.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420954325-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fb08cbe at consistency null
2022-03-22 06:10:17.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420954325-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:18.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040024403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ae09b7a at consistency null
2022-03-22 06:10:18.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040024403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:19.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348639116-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@749f8962 at consistency null
2022-03-22 06:10:19.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348639116-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:19.760 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:10:24.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112328953-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e1e3af3 at consistency null
2022-03-22 06:10:24.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112328953-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:24.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [116163783-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@651084f2 at consistency null
2022-03-22 06:10:24.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [116163783-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:25.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025876892-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b85f861 at consistency null
2022-03-22 06:10:25.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025876892-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:25.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2043671373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4359acea at consistency null
2022-03-22 06:10:25.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2043671373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:25.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576068495-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cdfdfd0 at consistency null
2022-03-22 06:10:25.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1576068495-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:25.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1062770762-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d09edad at consistency null
2022-03-22 06:10:25.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1062770762-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:25.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795993470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9ab569d at consistency null
2022-03-22 06:10:25.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795993470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:26.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [690105926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79b85be at consistency null
2022-03-22 06:10:26.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [690105926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:26.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931766384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75c8ebc2 at consistency null
2022-03-22 06:10:26.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1931766384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:27.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [138749590-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@578e1453 at consistency null
2022-03-22 06:10:27.444 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [138749590-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:28.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683632885-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@442af81 at consistency null
2022-03-22 06:10:28.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683632885-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:28.256 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:10:33.335 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [282538938-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5faf6cc2 at consistency null
2022-03-22 06:10:33.335 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [282538938-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:33.382 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081130042-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d8b0a0e at consistency null
2022-03-22 06:10:33.382 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081130042-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:33.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1629508404-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a00b760 at consistency null
2022-03-22 06:10:33.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1629508404-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:33.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [847996709-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b2f2706 at consistency null
2022-03-22 06:10:33.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [847996709-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:33.572 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [468240429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fc299dd at consistency null
2022-03-22 06:10:33.572 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [468240429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:33.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1589392902-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4aac3751 at consistency null
2022-03-22 06:10:33.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1589392902-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:33.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1030121238-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55822d3 at consistency null
2022-03-22 06:10:33.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1030121238-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:33.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1421708752-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c0a5b28 at consistency null
2022-03-22 06:10:33.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1421708752-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:34.087 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155819722-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67b7eb47 at consistency null
2022-03-22 06:10:34.087 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155819722-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:34.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [913999331-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23bd13d8 at consistency null
2022-03-22 06:10:34.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [913999331-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:34.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [950672109-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d8d93bc at consistency null
2022-03-22 06:10:34.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [950672109-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:35.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [4309797-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73c0463e at consistency null
2022-03-22 06:10:35.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [4309797-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:37.168 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1161540137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bf576d2 at consistency null
2022-03-22 06:10:37.168 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1161540137-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:37.173 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:10:42.185 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [655123184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@601d93a7 at consistency null
2022-03-22 06:10:42.185 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [655123184-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:42.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1327069582-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@618cfc6b at consistency null
2022-03-22 06:10:42.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1327069582-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:42.247 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:10:42.251 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:10:42.292 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1095750563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@100ca7d2 at consistency null
2022-03-22 06:10:42.292 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1095750563-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:42.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1509792601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@104f10e7 at consistency null
2022-03-22 06:10:42.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1509792601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:42.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [488798580-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c4329e2 at consistency null
2022-03-22 06:10:42.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [488798580-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:43.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [435398231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1aaae1a5 at consistency null
2022-03-22 06:10:43.048 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [435398231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:43.463 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [647937025-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f6df715 at consistency null
2022-03-22 06:10:43.463 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [647937025-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:44.272 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [327998680-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74d1a37 at consistency null
2022-03-22 06:10:44.273 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [327998680-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:45.087 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493616644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2443f05 at consistency null
2022-03-22 06:10:45.088 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493616644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:45.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [422458080-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f2e4121 at consistency null
2022-03-22 06:10:45.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [422458080-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:45.893 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:10:50.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1241774960-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ffbc28e at consistency null
2022-03-22 06:10:50.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1241774960-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:51.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067222260-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2088f5e4 at consistency null
2022-03-22 06:10:51.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1067222260-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:51.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2110117597-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3974ae7 at consistency null
2022-03-22 06:10:51.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2110117597-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:51.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [353653431-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ff37e10 at consistency null
2022-03-22 06:10:51.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [353653431-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:51.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083831070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@483399df at consistency null
2022-03-22 06:10:51.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1083831070-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:51.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [460940509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@714a224c at consistency null
2022-03-22 06:10:51.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [460940509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:52.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213803089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26b75a54 at consistency null
2022-03-22 06:10:52.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213803089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:52.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2137999367-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1280b36c at consistency null
2022-03-22 06:10:52.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2137999367-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:53.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893815799-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46a93d82 at consistency null
2022-03-22 06:10:53.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893815799-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:54.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [544114874-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66512f59 at consistency null
2022-03-22 06:10:54.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [544114874-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:54.622 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:10:59.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568603972-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a28ac3d at consistency null
2022-03-22 06:10:59.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568603972-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:59.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609119759-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6da47bc7 at consistency null
2022-03-22 06:10:59.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609119759-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:59.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1585109404-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62ee1edc at consistency null
2022-03-22 06:10:59.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1585109404-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:10:59.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [176533937-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@daade3f at consistency null
2022-03-22 06:10:59.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [176533937-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:00.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [181136304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@325cd2f3 at consistency null
2022-03-22 06:11:00.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [181136304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:00.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1562559216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c68fd3c at consistency null
2022-03-22 06:11:00.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1562559216-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:00.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436309233-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e620493 at consistency null
2022-03-22 06:11:00.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436309233-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:00.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1924365770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35626446 at consistency null
2022-03-22 06:11:00.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1924365770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:00.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [661691975-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61fdf353 at consistency null
2022-03-22 06:11:00.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [661691975-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:00.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821310235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ac7734d at consistency null
2022-03-22 06:11:00.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821310235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:00.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1536296357-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@247fa21a at consistency null
2022-03-22 06:11:00.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1536296357-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:01.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [849772080-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@299a287 at consistency null
2022-03-22 06:11:01.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [849772080-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:01.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [285695109-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cf0aeec at consistency null
2022-03-22 06:11:01.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [285695109-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:02.337 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [175775131-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6dcce924 at consistency null
2022-03-22 06:11:02.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [175775131-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:03.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091472988-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34faf9f5 at consistency null
2022-03-22 06:11:03.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091472988-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:03.215 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:11:08.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079525299-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bb8e877 at consistency null
2022-03-22 06:11:08.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079525299-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:08.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1804644425-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72b6be22 at consistency null
2022-03-22 06:11:08.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1804644425-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:08.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [836566629-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bef973a at consistency null
2022-03-22 06:11:08.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [836566629-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:08.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68545132-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27282f60 at consistency null
2022-03-22 06:11:08.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68545132-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:08.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [95549690-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60a8d209 at consistency null
2022-03-22 06:11:08.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [95549690-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:08.663 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1808597757-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57c237b1 at consistency null
2022-03-22 06:11:08.664 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1808597757-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:08.787 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026876983-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30733c34 at consistency null
2022-03-22 06:11:08.787 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026876983-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:08.999 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [374943095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56c64675 at consistency null
2022-03-22 06:11:08.999 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [374943095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:09.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472215918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3661202b at consistency null
2022-03-22 06:11:09.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472215918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:09.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [528237309-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f5a284f at consistency null
2022-03-22 06:11:09.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [528237309-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:10.479 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [98765435-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b4b203f at consistency null
2022-03-22 06:11:10.479 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [98765435-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:12.090 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1510911269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c550866 at consistency null
2022-03-22 06:11:12.090 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1510911269-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:12.092 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:11:12.272 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:11:12.279 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:11:17.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1838618891-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@730a19bf at consistency null
2022-03-22 06:11:17.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1838618891-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:17.151 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276940270-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d5c5fc1 at consistency null
2022-03-22 06:11:17.151 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276940270-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:17.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [221033815-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f34ca8d at consistency null
2022-03-22 06:11:17.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [221033815-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:17.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [585385554-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bc6e735 at consistency null
2022-03-22 06:11:17.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [585385554-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:17.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012109892-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5822de13 at consistency null
2022-03-22 06:11:17.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012109892-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:17.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776175586-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75f3081 at consistency null
2022-03-22 06:11:17.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776175586-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:17.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [28368771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e86f245 at consistency null
2022-03-22 06:11:17.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [28368771-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:17.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23519025-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72b555ef at consistency null
2022-03-22 06:11:17.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23519025-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:18.346 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74428197-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ca9cfec at consistency null
2022-03-22 06:11:18.346 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74428197-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:18.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [535127827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@428c3bc0 at consistency null
2022-03-22 06:11:18.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [535127827-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:19.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [405729195-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13d3d3ae at consistency null
2022-03-22 06:11:19.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [405729195-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:20.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [797324488-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61f7a897 at consistency null
2022-03-22 06:11:20.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [797324488-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:20.609 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:11:25.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1458155674-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4286dd52 at consistency null
2022-03-22 06:11:25.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1458155674-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:25.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644202155-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63bb181 at consistency null
2022-03-22 06:11:25.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644202155-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:25.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [230453495-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bf84c4b at consistency null
2022-03-22 06:11:25.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [230453495-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:26.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [318907900-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5717eeb3 at consistency null
2022-03-22 06:11:26.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [318907900-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:26.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [150223462-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@238fc54e at consistency null
2022-03-22 06:11:26.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [150223462-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:26.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851239939-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7863df07 at consistency null
2022-03-22 06:11:26.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851239939-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:26.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [8536310-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31304e54 at consistency null
2022-03-22 06:11:26.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [8536310-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:27.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [36546783-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59ba5cb7 at consistency null
2022-03-22 06:11:27.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [36546783-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:27.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732141046-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62bf2b03 at consistency null
2022-03-22 06:11:27.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732141046-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:28.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027356471-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@514e9bee at consistency null
2022-03-22 06:11:28.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027356471-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:29.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1782610731-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a1a2a94 at consistency null
2022-03-22 06:11:29.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1782610731-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:29.404 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:11:34.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96751084-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c75d723 at consistency null
2022-03-22 06:11:34.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96751084-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:34.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855602311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57890a9d at consistency null
2022-03-22 06:11:34.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [855602311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:34.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1927349590-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@723d7a69 at consistency null
2022-03-22 06:11:34.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1927349590-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:34.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [454144258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79f8b030 at consistency null
2022-03-22 06:11:34.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [454144258-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:34.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1008546977-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bd6ace3 at consistency null
2022-03-22 06:11:34.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1008546977-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:35.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1963930857-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41490e6 at consistency null
2022-03-22 06:11:35.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1963930857-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:35.647 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264131755-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d634c3f at consistency null
2022-03-22 06:11:35.648 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264131755-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:36.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431062527-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21261693 at consistency null
2022-03-22 06:11:36.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431062527-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:37.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281161812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f3907e0 at consistency null
2022-03-22 06:11:37.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281161812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:37.340 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:11:42.301 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:11:42.309 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:11:42.369 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081616202-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f0a3541 at consistency null
2022-03-22 06:11:42.369 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081616202-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:42.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025386722-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@288f962b at consistency null
2022-03-22 06:11:42.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025386722-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:42.531 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [802350139-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@398c8f27 at consistency null
2022-03-22 06:11:42.531 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [802350139-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:42.576 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83310031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e4d99f1 at consistency null
2022-03-22 06:11:42.576 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83310031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:42.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [95686764-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@130b44 at consistency null
2022-03-22 06:11:42.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [95686764-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:42.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235312896-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5479eba3 at consistency null
2022-03-22 06:11:42.736 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235312896-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:42.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1911207731-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@951b3bc at consistency null
2022-03-22 06:11:42.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1911207731-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:42.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1284418956-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bc12176 at consistency null
2022-03-22 06:11:42.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1284418956-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:42.981 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1771889409-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3904061 at consistency null
2022-03-22 06:11:42.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1771889409-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:43.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [737617491-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18af9743 at consistency null
2022-03-22 06:11:43.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [737617491-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:43.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [260067121-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ce70bb9 at consistency null
2022-03-22 06:11:43.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [260067121-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:43.799 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [911962496-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e3f43c2 at consistency null
2022-03-22 06:11:43.799 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [911962496-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:44.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [944834721-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f2d892c at consistency null
2022-03-22 06:11:44.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [944834721-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:45.051 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [713454662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b5c1205 at consistency null
2022-03-22 06:11:45.051 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [713454662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:45.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [670191584-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f426614 at consistency null
2022-03-22 06:11:45.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [670191584-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:45.863 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:11:50.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1246297583-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b170d6d at consistency null
2022-03-22 06:11:50.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1246297583-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:51.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2062348295-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@581097ca at consistency null
2022-03-22 06:11:51.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2062348295-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:51.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674682200-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53577af7 at consistency null
2022-03-22 06:11:51.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674682200-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:51.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1449742733-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ff9c44f at consistency null
2022-03-22 06:11:51.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1449742733-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:51.557 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531148586-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c86ccb8 at consistency null
2022-03-22 06:11:51.557 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531148586-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:52.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1177732749-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5074b425 at consistency null
2022-03-22 06:11:52.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1177732749-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:52.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [704031308-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@381cec1e at consistency null
2022-03-22 06:11:52.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [704031308-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:52.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890205126-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79868cee at consistency null
2022-03-22 06:11:52.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890205126-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:53.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1967511140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43df6c6a at consistency null
2022-03-22 06:11:53.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1967511140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:54.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622026991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@367f3da8 at consistency null
2022-03-22 06:11:54.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622026991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:54.316 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:11:59.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [959126219-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69dc4b65 at consistency null
2022-03-22 06:11:59.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [959126219-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:59.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012532587-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24705b4c at consistency null
2022-03-22 06:11:59.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012532587-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:59.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752468231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cf4df94 at consistency null
2022-03-22 06:11:59.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752468231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:59.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [317970539-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@532933df at consistency null
2022-03-22 06:11:59.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [317970539-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:11:59.946 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936684476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5825ddbc at consistency null
2022-03-22 06:11:59.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936684476-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:00.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [573666093-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e2957cf at consistency null
2022-03-22 06:12:00.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [573666093-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:00.635 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082285973-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@449b2fe1 at consistency null
2022-03-22 06:12:00.635 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082285973-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:01.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1287579502-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e364b66 at consistency null
2022-03-22 06:12:01.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1287579502-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:01.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1156889040-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2808a521 at consistency null
2022-03-22 06:12:01.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1156889040-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:02.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1844777592-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cab18c0 at consistency null
2022-03-22 06:12:02.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1844777592-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:02.626 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:12:07.651 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1071224484-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@152d893 at consistency null
2022-03-22 06:12:07.651 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1071224484-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:07.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82281382-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d53cea4 at consistency null
2022-03-22 06:12:07.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [82281382-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:07.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1706856659-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f4f82 at consistency null
2022-03-22 06:12:07.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1706856659-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:07.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [538917070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44363a59 at consistency null
2022-03-22 06:12:07.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [538917070-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:08.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1400956639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20e23620 at consistency null
2022-03-22 06:12:08.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1400956639-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:08.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1807922618-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a7b0ee1 at consistency null
2022-03-22 06:12:08.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1807922618-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:08.470 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794738713-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f6a0305 at consistency null
2022-03-22 06:12:08.470 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794738713-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:08.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [871962304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43119560 at consistency null
2022-03-22 06:12:08.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [871962304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:09.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1738254462-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7273f814 at consistency null
2022-03-22 06:12:09.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1738254462-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:11.372 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [428518994-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@707dfcbd at consistency null
2022-03-22 06:12:11.372 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [428518994-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:11.392 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:12:12.331 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:12:12.343 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:12:16.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065698872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f70ab9c at consistency null
2022-03-22 06:12:16.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065698872-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:16.488 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1152452638-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40058587 at consistency null
2022-03-22 06:12:16.488 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1152452638-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:16.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2127794350-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d5b76cb at consistency null
2022-03-22 06:12:16.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2127794350-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:16.739 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [845633816-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@325cffc9 at consistency null
2022-03-22 06:12:16.740 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [845633816-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:16.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [658927082-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38fa8e55 at consistency null
2022-03-22 06:12:16.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [658927082-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:17.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1811640548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2224c412 at consistency null
2022-03-22 06:12:17.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1811640548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:17.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156228565-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77d3d528 at consistency null
2022-03-22 06:12:17.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156228565-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:17.654 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [32042109-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bb775a1 at consistency null
2022-03-22 06:12:17.654 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [32042109-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:17.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [357955735-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@374f2f3a at consistency null
2022-03-22 06:12:17.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [357955735-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:18.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622446217-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@334f6995 at consistency null
2022-03-22 06:12:18.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [622446217-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:19.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1392455820-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5adc9041 at consistency null
2022-03-22 06:12:19.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1392455820-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:20.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [338863530-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ae6fcc6 at consistency null
2022-03-22 06:12:20.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [338863530-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:20.116 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:12:25.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [781805307-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@341e3a5f at consistency null
2022-03-22 06:12:25.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [781805307-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:25.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172295727-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cd5ee88 at consistency null
2022-03-22 06:12:25.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172295727-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:25.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1854923592-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@423527cb at consistency null
2022-03-22 06:12:25.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1854923592-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:25.557 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871779014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f78f387 at consistency null
2022-03-22 06:12:25.558 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871779014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:25.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [173676913-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4af1ecf5 at consistency null
2022-03-22 06:12:25.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [173676913-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:26.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [826277565-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72e38d09 at consistency null
2022-03-22 06:12:26.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [826277565-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:26.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540794481-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a4fac19 at consistency null
2022-03-22 06:12:26.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540794481-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:26.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1198246292-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f9c79e1 at consistency null
2022-03-22 06:12:26.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1198246292-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:26.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497074585-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79cdb30d at consistency null
2022-03-22 06:12:26.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497074585-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:26.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1394876038-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1728f2f8 at consistency null
2022-03-22 06:12:26.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1394876038-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:27.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [756189893-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5040bb60 at consistency null
2022-03-22 06:12:27.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [756189893-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:28.238 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [875352474-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c8f93f8 at consistency null
2022-03-22 06:12:28.239 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [875352474-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:29.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [798712463-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58ce04bf at consistency null
2022-03-22 06:12:29.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [798712463-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:29.112 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:12:34.140 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1547209172-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@821926c at consistency null
2022-03-22 06:12:34.140 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1547209172-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:34.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1699050548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5defa3b3 at consistency null
2022-03-22 06:12:34.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1699050548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:34.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [926402823-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d4ffba5 at consistency null
2022-03-22 06:12:34.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [926402823-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:34.633 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1137465972-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@450ce14f at consistency null
2022-03-22 06:12:34.633 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1137465972-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:34.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1537347056-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@413ab970 at consistency null
2022-03-22 06:12:34.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1537347056-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:35.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361248356-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@616878df at consistency null
2022-03-22 06:12:35.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361248356-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:35.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [478063139-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73e82584 at consistency null
2022-03-22 06:12:35.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [478063139-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:35.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1027623970-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@514ca42f at consistency null
2022-03-22 06:12:35.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1027623970-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:36.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1992690679-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4529664f at consistency null
2022-03-22 06:12:36.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1992690679-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:36.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [573122653-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@672550b5 at consistency null
2022-03-22 06:12:36.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [573122653-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:37.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273995657-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@610a3689 at consistency null
2022-03-22 06:12:37.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273995657-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:37.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1962005898-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f9f2ff9 at consistency null
2022-03-22 06:12:37.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1962005898-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:37.522 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:12:42.353 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:12:42.411 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:12:42.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [841937232-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22727a47 at consistency null
2022-03-22 06:12:42.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [841937232-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:42.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504599176-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b86dec4 at consistency null
2022-03-22 06:12:42.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504599176-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:42.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1870351837-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b881343 at consistency null
2022-03-22 06:12:42.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1870351837-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:42.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366774333-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dfe0f2c at consistency null
2022-03-22 06:12:42.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366774333-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:43.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1346199463-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cefe496 at consistency null
2022-03-22 06:12:43.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1346199463-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:43.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2078016308-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21c27531 at consistency null
2022-03-22 06:12:43.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2078016308-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:44.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1325182007-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a4b7841 at consistency null
2022-03-22 06:12:44.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1325182007-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:44.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1066420405-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@771974b at consistency null
2022-03-22 06:12:44.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1066420405-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:45.247 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [99281031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4391867 at consistency null
2022-03-22 06:12:45.247 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [99281031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:45.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [506764973-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36e08e2c at consistency null
2022-03-22 06:12:45.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [506764973-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:46.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897713224-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@570861c1 at consistency null
2022-03-22 06:12:46.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897713224-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:46.155 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:12:51.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [149563309-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cc87fa4 at consistency null
2022-03-22 06:12:51.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [149563309-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:51.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1519802599-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10febe7a at consistency null
2022-03-22 06:12:51.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1519802599-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:51.433 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1617786094-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22ddc3a7 at consistency null
2022-03-22 06:12:51.433 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1617786094-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:51.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1164479727-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@364fb854 at consistency null
2022-03-22 06:12:51.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1164479727-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:51.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484842656-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29e5d466 at consistency null
2022-03-22 06:12:51.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484842656-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:52.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1960783547-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c0da24e at consistency null
2022-03-22 06:12:52.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1960783547-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:53.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2141122250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@819093 at consistency null
2022-03-22 06:12:53.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2141122250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:55.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123368994-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b4705cf at consistency null
2022-03-22 06:12:55.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123368994-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:12:55.120 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:13:00.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1388251724-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c86ae25 at consistency null
2022-03-22 06:13:00.231 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1388251724-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:00.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603816534-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6db87a29 at consistency null
2022-03-22 06:13:00.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603816534-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:00.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1467486673-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61b16a05 at consistency null
2022-03-22 06:13:00.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1467486673-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:00.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [700956678-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12ab6281 at consistency null
2022-03-22 06:13:00.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [700956678-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:00.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [50348568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ba50177 at consistency null
2022-03-22 06:13:00.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [50348568-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:00.949 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510084306-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45e25718 at consistency null
2022-03-22 06:13:00.949 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510084306-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:01.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [151402582-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f69fce3 at consistency null
2022-03-22 06:13:01.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [151402582-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:01.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [517344896-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@661f2ce5 at consistency null
2022-03-22 06:13:01.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [517344896-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:02.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533176256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25f227e8 at consistency null
2022-03-22 06:13:02.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533176256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:03.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [992086334-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72fffed9 at consistency null
2022-03-22 06:13:03.241 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [992086334-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:03.243 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:13:08.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64901764-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12118a81 at consistency null
2022-03-22 06:13:08.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [64901764-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:08.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768798321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d76e422 at consistency null
2022-03-22 06:13:08.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768798321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:08.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [248008084-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@398f5f03 at consistency null
2022-03-22 06:13:08.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [248008084-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:08.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621338250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@348b77dd at consistency null
2022-03-22 06:13:08.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621338250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:08.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1412690653-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ad0a64f at consistency null
2022-03-22 06:13:08.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1412690653-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:09.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1848127308-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10bd10c3 at consistency null
2022-03-22 06:13:09.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1848127308-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:09.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [583035888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53093290 at consistency null
2022-03-22 06:13:09.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [583035888-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:09.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1105141164-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cd392b3 at consistency null
2022-03-22 06:13:09.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1105141164-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:10.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171031120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@131eb9b8 at consistency null
2022-03-22 06:13:10.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171031120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:10.568 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [625784341-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b5d9810 at consistency null
2022-03-22 06:13:10.568 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [625784341-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:11.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83023527-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71daaa59 at consistency null
2022-03-22 06:13:11.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83023527-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:11.441 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:13:12.413 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:13:12.437 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:13:16.467 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1039763944-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51bcf8db at consistency null
2022-03-22 06:13:16.467 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1039763944-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:16.545 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [974997309-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3709482d at consistency null
2022-03-22 06:13:16.546 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [974997309-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:16.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049028725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3093dbc4 at consistency null
2022-03-22 06:13:16.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2049028725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:16.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1752437880-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@143f27ed at consistency null
2022-03-22 06:13:16.893 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1752437880-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:17.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431187543-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22736f91 at consistency null
2022-03-22 06:13:17.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431187543-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:17.151 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1237839603-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fc613d2 at consistency null
2022-03-22 06:13:17.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1237839603-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:17.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1040338852-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71a5ae15 at consistency null
2022-03-22 06:13:17.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1040338852-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:17.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981631432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cc8ef83 at consistency null
2022-03-22 06:13:17.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981631432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:17.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1948125826-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25fe046c at consistency null
2022-03-22 06:13:17.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1948125826-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:17.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47760004-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ac9db82 at consistency null
2022-03-22 06:13:17.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47760004-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:17.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [780915220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@164001c8 at consistency null
2022-03-22 06:13:17.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [780915220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:18.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [852795142-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4daf4074 at consistency null
2022-03-22 06:13:18.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [852795142-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:18.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907725252-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18a7f0c9 at consistency null
2022-03-22 06:13:18.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907725252-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:19.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1913844065-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cb63d94 at consistency null
2022-03-22 06:13:19.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1913844065-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:20.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [41522757-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@aee218b at consistency null
2022-03-22 06:13:20.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [41522757-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:20.041 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:13:25.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [348329379-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d757c42 at consistency null
2022-03-22 06:13:25.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [348329379-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:25.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1962392086-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1af3f29d at consistency null
2022-03-22 06:13:25.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1962392086-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:25.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [294372368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@378436ca at consistency null
2022-03-22 06:13:25.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [294372368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:25.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603187674-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c9dabc8 at consistency null
2022-03-22 06:13:25.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603187674-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:25.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399638917-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a1c4975 at consistency null
2022-03-22 06:13:25.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399638917-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:26.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [436399348-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ffe2fb1 at consistency null
2022-03-22 06:13:26.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [436399348-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:26.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1441136864-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@683851fa at consistency null
2022-03-22 06:13:26.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1441136864-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:27.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136766949-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5057ea69 at consistency null
2022-03-22 06:13:27.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136766949-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:29.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1247393128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e7ac475 at consistency null
2022-03-22 06:13:29.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1247393128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:29.026 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:13:34.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1194427803-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64bde638 at consistency null
2022-03-22 06:13:34.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1194427803-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:34.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [582319261-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ea0dcd4 at consistency null
2022-03-22 06:13:34.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [582319261-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:34.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1044032127-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@214b63cb at consistency null
2022-03-22 06:13:34.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1044032127-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:34.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969914917-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@93a52fb at consistency null
2022-03-22 06:13:34.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969914917-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:34.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1922690522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23df9c2 at consistency null
2022-03-22 06:13:34.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1922690522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:35.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2076259968-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ab3d32d at consistency null
2022-03-22 06:13:35.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2076259968-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:36.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [281219797-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@309cb48b at consistency null
2022-03-22 06:13:36.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [281219797-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:36.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1410960506-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bea95e9 at consistency null
2022-03-22 06:13:36.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1410960506-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:36.919 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:13:41.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1775455262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67492bf at consistency null
2022-03-22 06:13:41.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1775455262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:41.997 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458490514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@528a13e7 at consistency null
2022-03-22 06:13:41.997 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458490514-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:42.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [327810711-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@767cdd1 at consistency null
2022-03-22 06:13:42.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [327810711-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:42.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [191617728-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@389d4509 at consistency null
2022-03-22 06:13:42.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [191617728-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:42.443 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:13:42.447 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:13:42.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2016134159-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a5ec9cf at consistency null
2022-03-22 06:13:42.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2016134159-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:42.954 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129322762-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1849f448 at consistency null
2022-03-22 06:13:42.954 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129322762-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:43.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [527510193-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21f9bbd5 at consistency null
2022-03-22 06:13:43.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [527510193-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:43.792 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243172421-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@739e17ee at consistency null
2022-03-22 06:13:43.792 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243172421-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:44.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1815404725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f62b8e3 at consistency null
2022-03-22 06:13:44.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1815404725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:45.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795815024-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48def4f3 at consistency null
2022-03-22 06:13:45.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [795815024-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:45.427 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:13:50.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [970829710-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2495d92e at consistency null
2022-03-22 06:13:50.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [970829710-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:50.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [968275350-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27378b35 at consistency null
2022-03-22 06:13:50.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [968275350-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:50.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1926028125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d188a90 at consistency null
2022-03-22 06:13:50.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1926028125-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:50.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613836248-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@125e8aac at consistency null
2022-03-22 06:13:50.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613836248-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:50.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1885540123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18086d60 at consistency null
2022-03-22 06:13:50.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1885540123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:51.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088046212-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fd3377 at consistency null
2022-03-22 06:13:51.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088046212-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:51.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1761031268-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ad87592 at consistency null
2022-03-22 06:13:51.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1761031268-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:51.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136482823-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cb5998f at consistency null
2022-03-22 06:13:51.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136482823-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:52.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397141792-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@570d9614 at consistency null
2022-03-22 06:13:52.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397141792-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:52.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1140775551-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d816109 at consistency null
2022-03-22 06:13:52.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1140775551-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:53.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310084208-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@145cf7ce at consistency null
2022-03-22 06:13:53.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [310084208-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:53.844 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:13:58.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [750642212-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34857d2b at consistency null
2022-03-22 06:13:58.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [750642212-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:58.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692894324-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f83f099 at consistency null
2022-03-22 06:13:58.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692894324-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:59.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1375700917-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@393c4149 at consistency null
2022-03-22 06:13:59.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1375700917-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:59.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [525985784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4aad3327 at consistency null
2022-03-22 06:13:59.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [525985784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:59.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552329971-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55de637c at consistency null
2022-03-22 06:13:59.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552329971-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:59.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68835123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45494666 at consistency null
2022-03-22 06:13:59.539 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68835123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:13:59.965 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695577130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e24718f at consistency null
2022-03-22 06:13:59.965 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695577130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:00.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2034495195-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3db34d67 at consistency null
2022-03-22 06:14:00.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2034495195-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:02.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [273364020-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b501c2e at consistency null
2022-03-22 06:14:02.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [273364020-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:02.418 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:14:07.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1186876097-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14250354 at consistency null
2022-03-22 06:14:07.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1186876097-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:07.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1109298374-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29b0b056 at consistency null
2022-03-22 06:14:07.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1109298374-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:07.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609060712-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e4f0872 at consistency null
2022-03-22 06:14:07.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609060712-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:07.584 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835990337-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15e5c4f9 at consistency null
2022-03-22 06:14:07.584 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835990337-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:07.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504856687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29664ab2 at consistency null
2022-03-22 06:14:07.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504856687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:07.757 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1870435619-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5787dc8 at consistency null
2022-03-22 06:14:07.757 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1870435619-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:07.863 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [658647395-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7382e98d at consistency null
2022-03-22 06:14:07.864 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [658647395-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:07.969 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1029463103-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48fb3908 at consistency null
2022-03-22 06:14:07.969 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1029463103-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:08.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1995872457-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53d3d97f at consistency null
2022-03-22 06:14:08.093 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1995872457-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:08.300 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2108219065-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@158378f0 at consistency null
2022-03-22 06:14:08.300 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2108219065-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:08.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1268246878-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71a9254c at consistency null
2022-03-22 06:14:08.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1268246878-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:09.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [343275959-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@600b6108 at consistency null
2022-03-22 06:14:09.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [343275959-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:09.565 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1443306272-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2dde4457 at consistency null
2022-03-22 06:14:09.565 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1443306272-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:10.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835096655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2314fde4 at consistency null
2022-03-22 06:14:10.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1835096655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:10.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [424186788-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1baf2f8f at consistency null
2022-03-22 06:14:10.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [424186788-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:10.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575371643-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20171ed3 at consistency null
2022-03-22 06:14:10.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575371643-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:10.853 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:14:12.477 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:14:12.517 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:14:15.861 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1291565065-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f341a7a at consistency null
2022-03-22 06:14:15.861 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1291565065-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:15.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724508923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bcb23a5 at consistency null
2022-03-22 06:14:15.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724508923-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:15.953 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982284681-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19fab804 at consistency null
2022-03-22 06:14:15.954 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982284681-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:16.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266239127-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@183dc6e at consistency null
2022-03-22 06:14:16.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266239127-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:16.085 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510961301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e3be1ee at consistency null
2022-03-22 06:14:16.086 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510961301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:16.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1276015960-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c4f2787 at consistency null
2022-03-22 06:14:16.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1276015960-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:16.285 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [16338011-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30b98a92 at consistency null
2022-03-22 06:14:16.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [16338011-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:16.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2092933655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f465047 at consistency null
2022-03-22 06:14:16.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2092933655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:16.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1195701019-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e73f0d0 at consistency null
2022-03-22 06:14:16.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1195701019-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:17.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027940382-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b059986 at consistency null
2022-03-22 06:14:17.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027940382-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:18.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839838410-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57946b6e at consistency null
2022-03-22 06:14:18.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839838410-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:19.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866146694-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a4dcdb3 at consistency null
2022-03-22 06:14:19.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866146694-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:19.619 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:14:24.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1394387138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b09415a at consistency null
2022-03-22 06:14:24.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1394387138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:24.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1410611109-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cc73e14 at consistency null
2022-03-22 06:14:24.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1410611109-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:24.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [285855575-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8fcf5f9 at consistency null
2022-03-22 06:14:24.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [285855575-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:25.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542431385-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cf88100 at consistency null
2022-03-22 06:14:25.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542431385-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:25.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [381169385-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d4a1654 at consistency null
2022-03-22 06:14:25.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [381169385-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:25.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667537540-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58b8ac77 at consistency null
2022-03-22 06:14:25.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667537540-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:26.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1019777736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3704c9a4 at consistency null
2022-03-22 06:14:26.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1019777736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:26.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [979240408-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f1caec5 at consistency null
2022-03-22 06:14:26.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [979240408-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:27.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1833351964-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1229ebc4 at consistency null
2022-03-22 06:14:27.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1833351964-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:28.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [716291210-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f6f36e2 at consistency null
2022-03-22 06:14:28.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [716291210-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:28.111 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:14:33.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1377252806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1eb79742 at consistency null
2022-03-22 06:14:33.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1377252806-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:33.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [405641881-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@246768d7 at consistency null
2022-03-22 06:14:33.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [405641881-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:33.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1089162459-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79cdd054 at consistency null
2022-03-22 06:14:33.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1089162459-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:33.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1729752480-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d4caa80 at consistency null
2022-03-22 06:14:33.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1729752480-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:33.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214654336-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32724e2e at consistency null
2022-03-22 06:14:33.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [214654336-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:33.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026305490-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25aa6744 at consistency null
2022-03-22 06:14:33.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026305490-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:33.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1657926021-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33d2f421 at consistency null
2022-03-22 06:14:33.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1657926021-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:33.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1242391698-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b025071 at consistency null
2022-03-22 06:14:33.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1242391698-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:34.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [477890318-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b216201 at consistency null
2022-03-22 06:14:34.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [477890318-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:34.140 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1405815604-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2dc7be66 at consistency null
2022-03-22 06:14:34.140 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1405815604-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:34.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1048282385-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9903403 at consistency null
2022-03-22 06:14:34.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1048282385-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:34.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536039352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60786c98 at consistency null
2022-03-22 06:14:34.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536039352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:34.635 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674280767-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30193bfc at consistency null
2022-03-22 06:14:34.635 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674280767-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:35.061 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1750223577-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@203d8717 at consistency null
2022-03-22 06:14:35.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1750223577-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:35.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [174266831-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28859f3f at consistency null
2022-03-22 06:14:35.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [174266831-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:36.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258480564-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b3a3a1b at consistency null
2022-03-22 06:14:36.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258480564-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:36.355 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:14:41.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [929218017-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7205265b at consistency null
2022-03-22 06:14:41.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [929218017-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:41.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1302080854-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ecc0233 at consistency null
2022-03-22 06:14:41.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1302080854-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:41.485 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596188747-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a341eb2 at consistency null
2022-03-22 06:14:41.485 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1596188747-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:41.592 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [273285026-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6004010d at consistency null
2022-03-22 06:14:41.592 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [273285026-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:41.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [845600255-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3be8c4a8 at consistency null
2022-03-22 06:14:41.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [845600255-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:41.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [507295864-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@656b947c at consistency null
2022-03-22 06:14:41.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [507295864-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:41.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578990148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fb0c7ed at consistency null
2022-03-22 06:14:41.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578990148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:42.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [24525514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ffcbd7f at consistency null
2022-03-22 06:14:42.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [24525514-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:42.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96198198-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bbd57fd at consistency null
2022-03-22 06:14:42.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [96198198-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:42.518 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:14:42.520 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:14:42.633 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2118080897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24b68c8 at consistency null
2022-03-22 06:14:42.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2118080897-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:43.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1271251800-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58062abb at consistency null
2022-03-22 06:14:43.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1271251800-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:43.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2077556864-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7df26394 at consistency null
2022-03-22 06:14:43.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2077556864-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:44.245 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541342626-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ce1c168 at consistency null
2022-03-22 06:14:44.245 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [541342626-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:44.247 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:14:49.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607362426-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3535f914 at consistency null
2022-03-22 06:14:49.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [607362426-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:49.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099519399-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c8f5efb at consistency null
2022-03-22 06:14:49.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099519399-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:49.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [302750742-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@320f683d at consistency null
2022-03-22 06:14:49.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [302750742-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:49.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [328416478-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50071595 at consistency null
2022-03-22 06:14:49.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [328416478-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:49.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [729004955-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a60bbd8 at consistency null
2022-03-22 06:14:49.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [729004955-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:50.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [551348031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cbe5ce9 at consistency null
2022-03-22 06:14:50.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [551348031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:50.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1861821470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25399fc3 at consistency null
2022-03-22 06:14:50.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1861821470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:51.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1686536403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dfa9d4c at consistency null
2022-03-22 06:14:51.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1686536403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:52.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1377783995-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28b4150 at consistency null
2022-03-22 06:14:52.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1377783995-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:52.615 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:14:57.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [356364504-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c4c0cfc at consistency null
2022-03-22 06:14:57.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [356364504-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:57.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851689397-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75e25f31 at consistency null
2022-03-22 06:14:57.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851689397-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:57.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1653809446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cebac3c at consistency null
2022-03-22 06:14:57.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1653809446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:57.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243688775-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3432f130 at consistency null
2022-03-22 06:14:57.847 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243688775-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:57.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [845348267-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48e520a6 at consistency null
2022-03-22 06:14:57.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [845348267-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:58.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [494129733-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f19e55c at consistency null
2022-03-22 06:14:58.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [494129733-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:58.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1720867548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e2236cf at consistency null
2022-03-22 06:14:58.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1720867548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:58.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621419102-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dbaff4a at consistency null
2022-03-22 06:14:58.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [621419102-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:58.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [10495217-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c3b084b at consistency null
2022-03-22 06:14:58.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [10495217-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:58.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [63057754-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33f3e899 at consistency null
2022-03-22 06:14:58.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [63057754-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:58.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2004013380-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59590102 at consistency null
2022-03-22 06:14:58.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2004013380-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:59.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [39845150-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5da43332 at consistency null
2022-03-22 06:14:59.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [39845150-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:59.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [56219144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@201d618e at consistency null
2022-03-22 06:14:59.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [56219144-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:14:59.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [301171142-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c06f234 at consistency null
2022-03-22 06:14:59.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [301171142-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:00.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886034509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d46a3a7 at consistency null
2022-03-22 06:15:00.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886034509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:01.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220216915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61072e9e at consistency null
2022-03-22 06:15:01.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220216915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:01.231 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:15:06.245 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082409065-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2add3221 at consistency null
2022-03-22 06:15:06.245 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1082409065-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:06.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1299448846-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11f1f21c at consistency null
2022-03-22 06:15:06.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1299448846-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:06.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1768441631-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3902201e at consistency null
2022-03-22 06:15:06.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1768441631-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:06.390 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1706438343-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61a98a7e at consistency null
2022-03-22 06:15:06.390 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1706438343-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:06.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [122928400-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2226e050 at consistency null
2022-03-22 06:15:06.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [122928400-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:06.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1793756544-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c59971a at consistency null
2022-03-22 06:15:06.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1793756544-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:06.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091661693-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1473425f at consistency null
2022-03-22 06:15:06.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1091661693-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:06.779 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 06:15:06.785 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-47, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 06:15:06.785 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-47, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 06:15:06.786 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-47, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 06:15:06.787 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-47, inFlight=0, closed=true] closing connection
2022-03-22 06:15:06.787 DEBUG 10 --- [uster-worker-85] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 06:15:06.789 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 06:15:06.789 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 600000 ms
2022-03-22 06:15:06.949 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361079182-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52ebf07b at consistency null
2022-03-22 06:15:06.949 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [361079182-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:07.370 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254031123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d56a09b at consistency null
2022-03-22 06:15:07.370 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254031123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:08.193 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [832319482-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dddb96b at consistency null
2022-03-22 06:15:08.194 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [832319482-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:08.997 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2075972659-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30a4641e at consistency null
2022-03-22 06:15:08.997 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2075972659-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:09.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1133351707-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d412259 at consistency null
2022-03-22 06:15:09.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1133351707-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:09.826 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:15:12.545 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:15:12.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:15:14.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1840233172-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45eb1f12 at consistency null
2022-03-22 06:15:14.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1840233172-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:14.870 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025293156-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@655de025 at consistency null
2022-03-22 06:15:14.871 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1025293156-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:14.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [538770872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76caec04 at consistency null
2022-03-22 06:15:14.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [538770872-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:14.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254195691-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@422d4b6e at consistency null
2022-03-22 06:15:14.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254195691-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:14.969 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533710273-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f23dfb6 at consistency null
2022-03-22 06:15:14.969 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533710273-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:14.997 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1196464839-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@557540bf at consistency null
2022-03-22 06:15:14.997 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1196464839-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:15.051 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15097512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@747e2cd9 at consistency null
2022-03-22 06:15:15.051 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15097512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:15.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985999536-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c207f03 at consistency null
2022-03-22 06:15:15.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985999536-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:15.368 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531253732-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dbfd55e at consistency null
2022-03-22 06:15:15.368 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531253732-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:15.572 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402664681-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62300eee at consistency null
2022-03-22 06:15:15.572 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402664681-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:15.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [706814559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@672dfe95 at consistency null
2022-03-22 06:15:15.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [706814559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:15.981 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [959581955-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@382a4d14 at consistency null
2022-03-22 06:15:15.981 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [959581955-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:16.473 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243345394-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b5c6a6a at consistency null
2022-03-22 06:15:16.473 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243345394-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:17.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399192380-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39f94891 at consistency null
2022-03-22 06:15:17.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399192380-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:17.310 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:15:22.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886722261-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3128c06d at consistency null
2022-03-22 06:15:22.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886722261-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:22.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1494633423-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b8069b1 at consistency null
2022-03-22 06:15:22.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1494633423-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:22.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1276622206-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63b74791 at consistency null
2022-03-22 06:15:22.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1276622206-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:22.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [819399643-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10f926b6 at consistency null
2022-03-22 06:15:22.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [819399643-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:22.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757159076-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67a5a76c at consistency null
2022-03-22 06:15:22.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757159076-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:22.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [52389967-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fb087c8 at consistency null
2022-03-22 06:15:22.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [52389967-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:23.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1926398231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6662d460 at consistency null
2022-03-22 06:15:23.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1926398231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:23.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052310341-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f77928c at consistency null
2022-03-22 06:15:23.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052310341-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:23.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1922215317-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75a3ccc9 at consistency null
2022-03-22 06:15:23.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1922215317-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:24.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179826328-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e168b72 at consistency null
2022-03-22 06:15:24.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179826328-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:24.939 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1812663520-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e279704 at consistency null
2022-03-22 06:15:24.939 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1812663520-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:24.942 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:15:30.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [135697865-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d4791be at consistency null
2022-03-22 06:15:30.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [135697865-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:30.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1863282523-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56e9d75b at consistency null
2022-03-22 06:15:30.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1863282523-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:30.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1517920012-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c5e8bef at consistency null
2022-03-22 06:15:30.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1517920012-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:30.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1847719087-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e43b6d7 at consistency null
2022-03-22 06:15:30.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1847719087-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:30.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1378196607-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f8818c0 at consistency null
2022-03-22 06:15:30.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1378196607-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:30.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1187267799-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37c34165 at consistency null
2022-03-22 06:15:30.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1187267799-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:30.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1356018428-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f9ab581 at consistency null
2022-03-22 06:15:30.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1356018428-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:30.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825630917-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c5c43ab at consistency null
2022-03-22 06:15:30.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825630917-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:30.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1326355704-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@337d1ef8 at consistency null
2022-03-22 06:15:30.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1326355704-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:31.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [730366770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e98b99b at consistency null
2022-03-22 06:15:31.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [730366770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:31.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1652887661-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24143174 at consistency null
2022-03-22 06:15:31.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1652887661-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:31.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1656038640-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1aeb7f46 at consistency null
2022-03-22 06:15:31.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1656038640-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:32.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495782666-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41ba283d at consistency null
2022-03-22 06:15:32.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495782666-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:32.522 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:15:37.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [920214543-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a3505f0 at consistency null
2022-03-22 06:15:37.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [920214543-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:37.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2090149057-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1faf0853 at consistency null
2022-03-22 06:15:37.559 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2090149057-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:37.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [936123655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5eff55af at consistency null
2022-03-22 06:15:37.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [936123655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:37.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344536945-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e14658d at consistency null
2022-03-22 06:15:37.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1344536945-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:37.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752944734-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cbcf33a at consistency null
2022-03-22 06:15:37.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [752944734-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:38.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038930479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b36916e at consistency null
2022-03-22 06:15:38.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2038930479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:38.358 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779387453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fc0fd8a at consistency null
2022-03-22 06:15:38.358 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [779387453-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:38.585 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1588273419-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5215cff2 at consistency null
2022-03-22 06:15:38.585 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1588273419-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:38.794 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052857718-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@492bd3a9 at consistency null
2022-03-22 06:15:38.794 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052857718-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:39.199 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1073709814-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66b99603 at consistency null
2022-03-22 06:15:39.199 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1073709814-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:39.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890276446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ead0bf3 at consistency null
2022-03-22 06:15:39.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890276446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:40.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [842286734-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22bb258b at consistency null
2022-03-22 06:15:40.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [842286734-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:40.425 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:15:40.545 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 06:15:40.562 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-71, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 06:15:40.562 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-71, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 06:15:40.562 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-71, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 06:15:40.580 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-71, inFlight=0, closed=true] closing connection
2022-03-22 06:15:40.581 DEBUG 10 --- [uster-worker-86] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 06:15:40.582 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 06:15:40.582 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 600000 ms
2022-03-22 06:15:42.565 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:15:42.595 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:15:45.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680085521-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@359a1cee at consistency null
2022-03-22 06:15:45.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [680085521-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:45.462 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1805817796-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fb99dd0 at consistency null
2022-03-22 06:15:45.462 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1805817796-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:45.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523692180-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29fdff47 at consistency null
2022-03-22 06:15:45.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523692180-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:45.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1245814646-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5211008e at consistency null
2022-03-22 06:15:45.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1245814646-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:45.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [686465007-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37890b4e at consistency null
2022-03-22 06:15:45.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [686465007-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:46.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1460220289-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@640d4772 at consistency null
2022-03-22 06:15:46.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1460220289-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:47.035 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@224c9b73 at consistency null
2022-03-22 06:15:47.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:47.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1189607868-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7803eac4 at consistency null
2022-03-22 06:15:47.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1189607868-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:47.919 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:15:53.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [982389608-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7af5df1 at consistency null
2022-03-22 06:15:53.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [982389608-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:53.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613791873-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@917aca1 at consistency null
2022-03-22 06:15:53.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613791873-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:53.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2133986489-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f4639f3 at consistency null
2022-03-22 06:15:53.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2133986489-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:53.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [115421434-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28f4bf15 at consistency null
2022-03-22 06:15:53.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [115421434-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:53.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1279157789-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55492f2f at consistency null
2022-03-22 06:15:53.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1279157789-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:53.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172737707-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@618fe383 at consistency null
2022-03-22 06:15:53.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172737707-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:53.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [710804842-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ff8c7e0 at consistency null
2022-03-22 06:15:53.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [710804842-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:54.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632784443-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ed5aa2a at consistency null
2022-03-22 06:15:54.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632784443-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:54.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [436605470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1058d388 at consistency null
2022-03-22 06:15:54.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [436605470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:54.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801737251-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49c6c1a4 at consistency null
2022-03-22 06:15:54.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1801737251-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:54.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [835414384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@151bfea6 at consistency null
2022-03-22 06:15:54.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [835414384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:54.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [676640556-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36bc82d5 at consistency null
2022-03-22 06:15:54.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [676640556-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:55.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [48809690-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@602ed319 at consistency null
2022-03-22 06:15:55.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [48809690-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:55.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273253864-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f1af559 at consistency null
2022-03-22 06:15:55.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273253864-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:56.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [554581773-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@553ec6d2 at consistency null
2022-03-22 06:15:56.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [554581773-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:15:56.611 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:16:01.645 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [745411031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7464c8a at consistency null
2022-03-22 06:16:01.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [745411031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:01.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1760877718-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11e6f160 at consistency null
2022-03-22 06:16:01.735 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1760877718-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:01.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [25576815-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@591e6e38 at consistency null
2022-03-22 06:16:01.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [25576815-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:01.952 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1369502725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ab52b9f at consistency null
2022-03-22 06:16:01.952 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1369502725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:02.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [219985743-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f221a5f at consistency null
2022-03-22 06:16:02.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [219985743-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:02.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136839190-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69e99f2a at consistency null
2022-03-22 06:16:02.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1136839190-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:02.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893483068-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43ac5381 at consistency null
2022-03-22 06:16:02.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893483068-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:03.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [646615294-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@109661b8 at consistency null
2022-03-22 06:16:03.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [646615294-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:03.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1321532847-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fb96117 at consistency null
2022-03-22 06:16:03.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1321532847-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:04.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461573147-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a8cb705 at consistency null
2022-03-22 06:16:04.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [461573147-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:04.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165437685-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b9dd18e at consistency null
2022-03-22 06:16:04.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165437685-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:04.919 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:16:09.952 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2074287184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6825e2e4 at consistency null
2022-03-22 06:16:09.953 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2074287184-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:10.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [308522766-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c10872e at consistency null
2022-03-22 06:16:10.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [308522766-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:10.084 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [293837453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@632b3136 at consistency null
2022-03-22 06:16:10.084 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [293837453-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:10.190 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066910080-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@469f9d7c at consistency null
2022-03-22 06:16:10.190 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066910080-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:10.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [412113194-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ea6589 at consistency null
2022-03-22 06:16:10.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [412113194-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:10.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394563305-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69abbb6d at consistency null
2022-03-22 06:16:10.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394563305-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:11.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [450771302-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53b395ef at consistency null
2022-03-22 06:16:11.230 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [450771302-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:11.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1969213238-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@697d8698 at consistency null
2022-03-22 06:16:11.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1969213238-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:12.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1864143294-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3be96013 at consistency null
2022-03-22 06:16:12.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1864143294-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:12.596 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:16:12.600 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:16:13.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549576562-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fadc4d0 at consistency null
2022-03-22 06:16:13.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549576562-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:13.252 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:16:18.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [925019798-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40fe3427 at consistency null
2022-03-22 06:16:18.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [925019798-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:18.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1016062267-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@453233e6 at consistency null
2022-03-22 06:16:18.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1016062267-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:18.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [932517493-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35ab3aab at consistency null
2022-03-22 06:16:18.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [932517493-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:18.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [247148236-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5441e38a at consistency null
2022-03-22 06:16:18.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [247148236-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:18.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743405459-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65afaa29 at consistency null
2022-03-22 06:16:18.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743405459-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:18.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871367448-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31560879 at consistency null
2022-03-22 06:16:18.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871367448-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:19.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1890801698-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dc4129a at consistency null
2022-03-22 06:16:19.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1890801698-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:19.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [657053110-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@642e71b at consistency null
2022-03-22 06:16:19.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [657053110-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:19.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1888341295-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ea4fb18 at consistency null
2022-03-22 06:16:19.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1888341295-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:19.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1230384171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f302d75 at consistency null
2022-03-22 06:16:19.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1230384171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:19.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [600711019-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@629daccb at consistency null
2022-03-22 06:16:19.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [600711019-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:20.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243867841-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73f6e5bc at consistency null
2022-03-22 06:16:20.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1243867841-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:20.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1423548959-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ab3ace1 at consistency null
2022-03-22 06:16:20.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1423548959-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:21.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [739703771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54f67b47 at consistency null
2022-03-22 06:16:21.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [739703771-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:21.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146471029-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b827ee6 at consistency null
2022-03-22 06:16:21.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146471029-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:21.845 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:435)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:16:27.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [709887144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f9dbb2f at consistency null
2022-03-22 06:16:27.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [709887144-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:27.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [864064167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@489ec899 at consistency null
2022-03-22 06:16:27.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [864064167-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:27.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [684949295-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f55be8e at consistency null
2022-03-22 06:16:27.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [684949295-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:27.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396432228-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67e880cc at consistency null
2022-03-22 06:16:27.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396432228-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:27.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045048559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@155e1bdd at consistency null
2022-03-22 06:16:27.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045048559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:28.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1321617118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bbac3ef at consistency null
2022-03-22 06:16:28.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1321617118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:28.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2111791662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2999a9d8 at consistency null
2022-03-22 06:16:28.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2111791662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:29.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1879195660-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ad4b2c7 at consistency null
2022-03-22 06:16:29.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1879195660-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:30.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [772772717-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fb1c0b3 at consistency null
2022-03-22 06:16:30.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [772772717-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:30.211 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:16:35.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1384370372-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12e9c311 at consistency null
2022-03-22 06:16:35.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1384370372-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:35.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1422591656-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@765a7ffd at consistency null
2022-03-22 06:16:35.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1422591656-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:35.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917192221-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59b7b97b at consistency null
2022-03-22 06:16:35.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917192221-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:35.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1684883294-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38624ea6 at consistency null
2022-03-22 06:16:35.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1684883294-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:35.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [673492711-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64cfe3f0 at consistency null
2022-03-22 06:16:35.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [673492711-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:36.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609616653-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41cfcd03 at consistency null
2022-03-22 06:16:36.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609616653-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:36.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273137740-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25d7d820 at consistency null
2022-03-22 06:16:36.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273137740-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:37.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1161524185-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f6ab542 at consistency null
2022-03-22 06:16:37.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1161524185-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:38.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223015951-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1421173c at consistency null
2022-03-22 06:16:38.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223015951-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:39.151 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1547666037-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bb9676e at consistency null
2022-03-22 06:16:39.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1547666037-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:39.154 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:16:42.626 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:16:42.633 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:16:44.174 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [55628628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3034ace1 at consistency null
2022-03-22 06:16:44.174 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [55628628-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:44.239 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493608284-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a2edf00 at consistency null
2022-03-22 06:16:44.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493608284-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:44.394 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2023427384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16c82622 at consistency null
2022-03-22 06:16:44.394 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2023427384-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:44.598 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1477907572-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38ec3be at consistency null
2022-03-22 06:16:44.598 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1477907572-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:45.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1574479441-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f44da10 at consistency null
2022-03-22 06:16:45.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1574479441-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:45.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [505024907-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c2142bb at consistency null
2022-03-22 06:16:45.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [505024907-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:45.869 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1053306301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b0f9a2b at consistency null
2022-03-22 06:16:45.870 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1053306301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:46.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587292836-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e6937c7 at consistency null
2022-03-22 06:16:46.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1587292836-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:46.743 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1190065416-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2744ea40 at consistency null
2022-03-22 06:16:46.743 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1190065416-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:47.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [146154007-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11c64089 at consistency null
2022-03-22 06:16:47.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [146154007-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:47.625 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:16:52.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2003587961-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60f4e58 at consistency null
2022-03-22 06:16:52.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2003587961-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:52.794 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1218088506-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@510f8651 at consistency null
2022-03-22 06:16:52.794 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1218088506-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:52.856 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045600916-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@511970da at consistency null
2022-03-22 06:16:52.857 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045600916-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:52.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081231140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7278b078 at consistency null
2022-03-22 06:16:52.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081231140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:53.035 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1427125192-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ef95de1 at consistency null
2022-03-22 06:16:53.035 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1427125192-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:53.134 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1239911438-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23e96f61 at consistency null
2022-03-22 06:16:53.134 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1239911438-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:53.239 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1539082338-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39ec5466 at consistency null
2022-03-22 06:16:53.239 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1539082338-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:53.393 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [161857366-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e20c853 at consistency null
2022-03-22 06:16:53.393 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [161857366-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:53.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1577014233-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4af29a23 at consistency null
2022-03-22 06:16:53.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1577014233-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:53.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671949716-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2208667b at consistency null
2022-03-22 06:16:53.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671949716-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:54.236 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [399327599-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@799c4938 at consistency null
2022-03-22 06:16:54.237 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [399327599-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:55.049 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859182031-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71766408 at consistency null
2022-03-22 06:16:55.049 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859182031-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:16:55.051 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:17:00.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [351338475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f8bcb87 at consistency null
2022-03-22 06:17:00.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [351338475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:00.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436577634-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ed15bb3 at consistency null
2022-03-22 06:17:00.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1436577634-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:00.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [672897664-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@551f1ed8 at consistency null
2022-03-22 06:17:00.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [672897664-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:00.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732978215-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@178b7657 at consistency null
2022-03-22 06:17:00.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732978215-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:00.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [942469405-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66ad50d2 at consistency null
2022-03-22 06:17:00.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [942469405-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:00.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1856061957-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22e58377 at consistency null
2022-03-22 06:17:00.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1856061957-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:01.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [446302923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f21a02d at consistency null
2022-03-22 06:17:01.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [446302923-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:01.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862076123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43544650 at consistency null
2022-03-22 06:17:01.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862076123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:02.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1813616563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ff3b528 at consistency null
2022-03-22 06:17:02.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1813616563-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:02.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139215199-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4183799a at consistency null
2022-03-22 06:17:02.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139215199-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:02.936 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:17:08.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112198863-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4321894f at consistency null
2022-03-22 06:17:08.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1112198863-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1617733305-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@222621bd at consistency null
2022-03-22 06:17:08.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1617733305-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1080561928-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70ae8a24 at consistency null
2022-03-22 06:17:08.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1080561928-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [717547235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a3b7202 at consistency null
2022-03-22 06:17:08.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [717547235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1055685058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@651fedfb at consistency null
2022-03-22 06:17:08.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1055685058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.339 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [993024456-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7627032c at consistency null
2022-03-22 06:17:08.339 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [993024456-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1003548663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@725c5866 at consistency null
2022-03-22 06:17:08.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1003548663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.469 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917431696-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3068f4d at consistency null
2022-03-22 06:17:08.469 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917431696-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1800517870-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19eccae3 at consistency null
2022-03-22 06:17:08.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1800517870-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1377194581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@579c8491 at consistency null
2022-03-22 06:17:08.567 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1377194581-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845866679-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4713696c at consistency null
2022-03-22 06:17:08.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845866679-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [793139842-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7058ea24 at consistency null
2022-03-22 06:17:08.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [793139842-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.763 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1228280140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f1c9d1e at consistency null
2022-03-22 06:17:08.763 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1228280140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:08.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [701654725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c972860 at consistency null
2022-03-22 06:17:08.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [701654725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:09.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1779559232-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60937e97 at consistency null
2022-03-22 06:17:09.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1779559232-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:09.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1998470858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57adb16b at consistency null
2022-03-22 06:17:09.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1998470858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:09.787 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [240905492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cb222ed at consistency null
2022-03-22 06:17:09.787 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [240905492-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:10.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [872760916-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39e3c933 at consistency null
2022-03-22 06:17:10.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [872760916-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:10.612 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:17:12.653 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:17:12.656 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:17:15.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1370222719-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@427f6f5e at consistency null
2022-03-22 06:17:15.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1370222719-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:15.699 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1464171758-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7863b893 at consistency null
2022-03-22 06:17:15.699 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1464171758-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:15.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [111609951-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55eb4838 at consistency null
2022-03-22 06:17:15.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [111609951-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:16.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [403622201-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7266a9e7 at consistency null
2022-03-22 06:17:16.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [403622201-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:16.459 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371780175-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b5a2d6a at consistency null
2022-03-22 06:17:16.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371780175-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:17.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794524776-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e18b3b2 at consistency null
2022-03-22 06:17:17.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794524776-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:18.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420648089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@213d94c1 at consistency null
2022-03-22 06:17:18.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420648089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:18.931 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:17:24.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981729844-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66d33890 at consistency null
2022-03-22 06:17:24.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981729844-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:24.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806695865-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cf8785d at consistency null
2022-03-22 06:17:24.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806695865-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:24.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [867312546-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1796d153 at consistency null
2022-03-22 06:17:24.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [867312546-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:24.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [223734491-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a40c4fc at consistency null
2022-03-22 06:17:24.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [223734491-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:24.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391826663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25a8709c at consistency null
2022-03-22 06:17:24.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391826663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:24.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [59554176-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a2ddee4 at consistency null
2022-03-22 06:17:24.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [59554176-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:25.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [562115936-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b1098bb at consistency null
2022-03-22 06:17:25.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [562115936-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:26.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60817593-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67abcc01 at consistency null
2022-03-22 06:17:26.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [60817593-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:27.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1113377246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e17cf8 at consistency null
2022-03-22 06:17:27.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1113377246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:27.720 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:17:32.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [620374188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75cf58de at consistency null
2022-03-22 06:17:32.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [620374188-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:32.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [217592678-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a25d39c at consistency null
2022-03-22 06:17:32.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [217592678-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:33.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257843080-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73d7c51d at consistency null
2022-03-22 06:17:33.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257843080-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:33.143 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [728266116-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79db6123 at consistency null
2022-03-22 06:17:33.143 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [728266116-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:33.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579321101-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@765ddf6e at consistency null
2022-03-22 06:17:33.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579321101-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:33.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [328006828-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fb29b7a at consistency null
2022-03-22 06:17:33.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [328006828-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:34.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1859339082-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d422bd6 at consistency null
2022-03-22 06:17:34.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1859339082-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:34.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109039425-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@657c3dbf at consistency null
2022-03-22 06:17:34.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109039425-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:34.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286548076-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7534e211 at consistency null
2022-03-22 06:17:34.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286548076-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:35.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [951244489-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e16de86 at consistency null
2022-03-22 06:17:35.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [951244489-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:36.054 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1276054406-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52e64e09 at consistency null
2022-03-22 06:17:36.054 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1276054406-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:36.055 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:17:41.061 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879935617-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56f3102f at consistency null
2022-03-22 06:17:41.061 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879935617-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:41.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [29061161-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@485448a at consistency null
2022-03-22 06:17:41.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [29061161-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:41.175 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789703241-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c6f5872 at consistency null
2022-03-22 06:17:41.176 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789703241-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:41.234 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [460756758-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c99d9f0 at consistency null
2022-03-22 06:17:41.234 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [460756758-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:41.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [966933041-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22a5de24 at consistency null
2022-03-22 06:17:41.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [966933041-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:41.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1665285908-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@caa7fee at consistency null
2022-03-22 06:17:41.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1665285908-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:41.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1126285975-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7af01ac1 at consistency null
2022-03-22 06:17:41.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1126285975-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:42.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1824535490-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58840a36 at consistency null
2022-03-22 06:17:42.071 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1824535490-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:42.483 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151070169-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@545cbbfc at consistency null
2022-03-22 06:17:42.484 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151070169-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:42.681 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:17:42.684 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:17:43.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [919851713-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b56eb96 at consistency null
2022-03-22 06:17:43.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [919851713-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:44.155 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066455542-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@710ade2f at consistency null
2022-03-22 06:17:44.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066455542-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:44.157 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:17:49.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [379124082-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@927bf82 at consistency null
2022-03-22 06:17:49.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [379124082-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:49.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [224803167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38157601 at consistency null
2022-03-22 06:17:49.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [224803167-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:49.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629909480-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a320ce at consistency null
2022-03-22 06:17:49.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629909480-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:49.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47737988-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@237f6dd3 at consistency null
2022-03-22 06:17:49.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47737988-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:49.577 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1398429380-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17100d91 at consistency null
2022-03-22 06:17:49.577 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1398429380-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:49.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1875517745-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@426d40c at consistency null
2022-03-22 06:17:49.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1875517745-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:49.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394686407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@191e3116 at consistency null
2022-03-22 06:17:49.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394686407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:50.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1459330559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4157d613 at consistency null
2022-03-22 06:17:50.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1459330559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:50.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1461067154-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56bc272d at consistency null
2022-03-22 06:17:50.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1461067154-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:50.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1768648168-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64880f42 at consistency null
2022-03-22 06:17:50.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1768648168-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:50.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1303891797-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cf61444 at consistency null
2022-03-22 06:17:50.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1303891797-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:51.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202304889-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73eeac87 at consistency null
2022-03-22 06:17:51.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202304889-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:51.868 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:17:57.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540330131-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@445c6898 at consistency null
2022-03-22 06:17:57.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540330131-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:57.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [302179664-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ce2e12f at consistency null
2022-03-22 06:17:57.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [302179664-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:57.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1631341858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a6ef4a0 at consistency null
2022-03-22 06:17:57.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1631341858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:57.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797536771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dd14fdd at consistency null
2022-03-22 06:17:57.343 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1797536771-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:57.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1842479524-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49054dd4 at consistency null
2022-03-22 06:17:57.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1842479524-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:57.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1761558827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a52642 at consistency null
2022-03-22 06:17:57.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1761558827-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:58.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806255179-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f430942 at consistency null
2022-03-22 06:17:58.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806255179-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:58.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [564635929-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@589b14d0 at consistency null
2022-03-22 06:17:58.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [564635929-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:58.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552237480-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b1389fa at consistency null
2022-03-22 06:17:58.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1552237480-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:17:59.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1323411210-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@102f2fec at consistency null
2022-03-22 06:17:59.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1323411210-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:00.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1867196801-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27912db2 at consistency null
2022-03-22 06:18:00.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1867196801-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:00.212 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:18:05.236 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1933097488-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27ac8004 at consistency null
2022-03-22 06:18:05.236 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1933097488-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:05.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [97923036-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@578152 at consistency null
2022-03-22 06:18:05.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [97923036-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:05.456 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1187774875-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69ee8efb at consistency null
2022-03-22 06:18:05.456 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1187774875-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:05.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887256546-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47bcc2c9 at consistency null
2022-03-22 06:18:05.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887256546-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:05.870 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176757191-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@453d69a at consistency null
2022-03-22 06:18:05.870 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176757191-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:06.441 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1916771344-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38328a53 at consistency null
2022-03-22 06:18:06.441 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1916771344-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:06.939 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415154453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25b652f8 at consistency null
2022-03-22 06:18:06.939 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415154453-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:07.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1296186451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43d660d at consistency null
2022-03-22 06:18:07.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1296186451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:07.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [346477543-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ec09716 at consistency null
2022-03-22 06:18:07.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [346477543-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:08.752 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [452395647-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@328cf68f at consistency null
2022-03-22 06:18:08.752 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [452395647-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:08.753 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:18:12.720 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:18:12.726 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:18:13.796 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49160983-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25ba53b4 at consistency null
2022-03-22 06:18:13.796 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49160983-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:13.873 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578248125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ad84400 at consistency null
2022-03-22 06:18:13.873 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578248125-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:13.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2089418355-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@503a026a at consistency null
2022-03-22 06:18:13.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2089418355-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:14.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1590689639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d7be6c at consistency null
2022-03-22 06:18:14.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1590689639-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:14.068 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [759863602-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72877a91 at consistency null
2022-03-22 06:18:14.068 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [759863602-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:14.136 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1655340265-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a77d885 at consistency null
2022-03-22 06:18:14.136 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1655340265-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:14.260 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1365474473-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7adebcf3 at consistency null
2022-03-22 06:18:14.260 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1365474473-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:14.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886815525-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42c2491a at consistency null
2022-03-22 06:18:14.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886815525-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:14.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [160170256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b58e021 at consistency null
2022-03-22 06:18:14.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [160170256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:14.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [946895356-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fafb23f at consistency null
2022-03-22 06:18:14.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [946895356-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:15.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1924979823-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4019b61d at consistency null
2022-03-22 06:18:15.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1924979823-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:15.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [908538032-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10c3ad9d at consistency null
2022-03-22 06:18:15.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [908538032-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:16.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1607087077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cd61e4c at consistency null
2022-03-22 06:18:16.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1607087077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:16.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1682841209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e5a61a3 at consistency null
2022-03-22 06:18:16.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1682841209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:17.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1879435579-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@637e8b2b at consistency null
2022-03-22 06:18:17.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1879435579-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:17.713 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:18:22.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2141779259-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6edc6d9c at consistency null
2022-03-22 06:18:22.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2141779259-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:22.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1911796784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2143a75 at consistency null
2022-03-22 06:18:22.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1911796784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:23.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1700450721-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f7e04ed at consistency null
2022-03-22 06:18:23.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1700450721-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:23.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [977903616-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28cee7fa at consistency null
2022-03-22 06:18:23.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [977903616-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:23.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [40930143-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25f3dff2 at consistency null
2022-03-22 06:18:23.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [40930143-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:23.962 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1859054869-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6179eab at consistency null
2022-03-22 06:18:23.962 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1859054869-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:24.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571487412-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4901bbe8 at consistency null
2022-03-22 06:18:24.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1571487412-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:24.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [714349729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16af6405 at consistency null
2022-03-22 06:18:24.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [714349729-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:25.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [741884127-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d98356d at consistency null
2022-03-22 06:18:25.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [741884127-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:25.624 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:18:30.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889697800-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@174004b3 at consistency null
2022-03-22 06:18:30.728 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889697800-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:30.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1062383938-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@535a3348 at consistency null
2022-03-22 06:18:30.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1062383938-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:31.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [900322077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e38fa6a at consistency null
2022-03-22 06:18:31.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [900322077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:31.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1543062479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5267839d at consistency null
2022-03-22 06:18:31.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1543062479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:31.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1397102926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@459560fe at consistency null
2022-03-22 06:18:31.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1397102926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:32.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2033482437-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47d7db77 at consistency null
2022-03-22 06:18:32.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2033482437-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:33.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [50233285-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e45824d at consistency null
2022-03-22 06:18:33.157 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [50233285-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:33.171 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:18:38.253 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [319337326-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50d08573 at consistency null
2022-03-22 06:18:38.253 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [319337326-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:38.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1885815848-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3eac8d07 at consistency null
2022-03-22 06:18:38.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1885815848-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:38.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [710933171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47d9f5c6 at consistency null
2022-03-22 06:18:38.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [710933171-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:38.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1529788522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@551c509a at consistency null
2022-03-22 06:18:38.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1529788522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:38.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [225178530-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@133bc57 at consistency null
2022-03-22 06:18:38.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [225178530-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:38.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197977635-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b2dddc3 at consistency null
2022-03-22 06:18:38.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197977635-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:38.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [314069022-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@535264a5 at consistency null
2022-03-22 06:18:38.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [314069022-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:39.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1023104971-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@647d76d5 at consistency null
2022-03-22 06:18:39.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1023104971-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:39.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851741461-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78b0bf09 at consistency null
2022-03-22 06:18:39.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851741461-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:39.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [348325461-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d46a619 at consistency null
2022-03-22 06:18:39.635 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [348325461-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:39.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1143704691-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23590ed at consistency null
2022-03-22 06:18:39.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1143704691-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:40.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [571002418-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f002f55 at consistency null
2022-03-22 06:18:40.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [571002418-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:41.140 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921419931-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78e38c2 at consistency null
2022-03-22 06:18:41.151 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921419931-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:42.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [982948285-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6dcbf2f at consistency null
2022-03-22 06:18:42.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [982948285-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:42.028 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:18:42.749 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:18:42.891 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:18:47.058 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [583455368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28c0f5e1 at consistency null
2022-03-22 06:18:47.058 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [583455368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:47.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1188815292-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56c41d2b at consistency null
2022-03-22 06:18:47.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1188815292-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:47.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [777785060-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@207a6719 at consistency null
2022-03-22 06:18:47.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [777785060-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:47.341 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [537473507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43bc81fe at consistency null
2022-03-22 06:18:47.341 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [537473507-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:47.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [886040056-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c7c0b95 at consistency null
2022-03-22 06:18:47.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [886040056-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:47.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754988252-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1aa76e74 at consistency null
2022-03-22 06:18:47.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754988252-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:48.056 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667577014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69169096 at consistency null
2022-03-22 06:18:48.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667577014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:48.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [168466192-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e825f3b at consistency null
2022-03-22 06:18:48.334 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [168466192-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:48.758 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1101087628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@422ed5dd at consistency null
2022-03-22 06:18:48.758 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1101087628-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:49.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1831545264-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@635bd722 at consistency null
2022-03-22 06:18:49.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1831545264-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:49.613 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:18:54.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [664891431-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e1469ff at consistency null
2022-03-22 06:18:54.646 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [664891431-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:54.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2043527636-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62338281 at consistency null
2022-03-22 06:18:54.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2043527636-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:55.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851136578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56072327 at consistency null
2022-03-22 06:18:55.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851136578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:55.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [107579660-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b580c02 at consistency null
2022-03-22 06:18:55.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [107579660-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:55.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [719708590-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@761edb07 at consistency null
2022-03-22 06:18:55.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [719708590-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:55.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026643527-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@84768cf at consistency null
2022-03-22 06:18:55.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026643527-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:56.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1432369379-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67367cfb at consistency null
2022-03-22 06:18:56.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1432369379-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:56.858 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [520874666-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c254403 at consistency null
2022-03-22 06:18:56.858 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [520874666-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:57.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [442925172-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ce37863 at consistency null
2022-03-22 06:18:57.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [442925172-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:58.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768765457-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f517020 at consistency null
2022-03-22 06:18:58.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [768765457-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:18:58.613 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:19:03.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040654952-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bd40a95 at consistency null
2022-03-22 06:19:03.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040654952-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:03.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [960436044-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d3756b0 at consistency null
2022-03-22 06:19:03.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [960436044-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:03.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [393499967-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ac00f15 at consistency null
2022-03-22 06:19:03.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [393499967-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:04.061 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [694686502-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ee4ac1d at consistency null
2022-03-22 06:19:04.061 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [694686502-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:04.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599891663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65ef93f6 at consistency null
2022-03-22 06:19:04.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599891663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:04.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504926495-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@121acb2d at consistency null
2022-03-22 06:19:04.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504926495-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:04.962 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [728822350-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8b8d6dc at consistency null
2022-03-22 06:19:04.962 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [728822350-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:05.955 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1114121509-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43f71792 at consistency null
2022-03-22 06:19:05.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1114121509-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:07.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2127443895-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a9e5c39 at consistency null
2022-03-22 06:19:07.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2127443895-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:07.621 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:19:12.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [604291352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e19a7e4 at consistency null
2022-03-22 06:19:12.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [604291352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:12.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139109716-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50d99ccd at consistency null
2022-03-22 06:19:12.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139109716-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:12.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [193525877-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2be94ac3 at consistency null
2022-03-22 06:19:12.811 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [193525877-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:12.899 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:19:12.919 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:19:12.937 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83100988-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3beb413 at consistency null
2022-03-22 06:19:12.937 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [83100988-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:13.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [117049531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@504b0507 at consistency null
2022-03-22 06:19:13.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [117049531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:13.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531407669-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c68c437 at consistency null
2022-03-22 06:19:13.354 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1531407669-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:13.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390098296-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d36202b at consistency null
2022-03-22 06:19:13.603 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [390098296-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:13.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859391817-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b5e07b2 at consistency null
2022-03-22 06:19:13.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859391817-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:14.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1699753541-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@533542e3 at consistency null
2022-03-22 06:19:14.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1699753541-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:15.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [94296351-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59766dce at consistency null
2022-03-22 06:19:15.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [94296351-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:15.865 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [287399801-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@455d39b3 at consistency null
2022-03-22 06:19:15.865 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [287399801-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:15.870 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:19:21.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1759199515-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e15d53d at consistency null
2022-03-22 06:19:21.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1759199515-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:21.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [211642435-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@174eb046 at consistency null
2022-03-22 06:19:21.117 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [211642435-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:21.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2134384366-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d3ca77c at consistency null
2022-03-22 06:19:21.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2134384366-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:21.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1685671143-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@518089c9 at consistency null
2022-03-22 06:19:21.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1685671143-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:21.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1444416234-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73524c10 at consistency null
2022-03-22 06:19:21.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1444416234-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:21.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [824229658-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b4d0241 at consistency null
2022-03-22 06:19:21.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [824229658-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:21.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [248518846-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e560020 at consistency null
2022-03-22 06:19:21.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [248518846-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:22.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [543192055-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fcc26c9 at consistency null
2022-03-22 06:19:22.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [543192055-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:22.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [909734475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e1fc1ec at consistency null
2022-03-22 06:19:22.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [909734475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:22.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47567865-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d439164 at consistency null
2022-03-22 06:19:22.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47567865-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:23.265 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [206454517-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@231ac3ec at consistency null
2022-03-22 06:19:23.265 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [206454517-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:24.237 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789711680-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d4e5328 at consistency null
2022-03-22 06:19:24.237 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789711680-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:24.239 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:19:29.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1388617667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d8ef3cc at consistency null
2022-03-22 06:19:29.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1388617667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:29.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2124600441-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c7781a8 at consistency null
2022-03-22 06:19:29.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2124600441-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:29.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [57074894-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19872a69 at consistency null
2022-03-22 06:19:29.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [57074894-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:29.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1007794153-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66affd62 at consistency null
2022-03-22 06:19:29.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1007794153-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:30.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256004911-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c9572e8 at consistency null
2022-03-22 06:19:30.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256004911-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:30.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [573835537-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c3fc013 at consistency null
2022-03-22 06:19:30.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [573835537-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:30.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [381733133-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c4b659f at consistency null
2022-03-22 06:19:30.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [381733133-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:30.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [660576963-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2516cb99 at consistency null
2022-03-22 06:19:30.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [660576963-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:31.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [97111906-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42f3ffe1 at consistency null
2022-03-22 06:19:31.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [97111906-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:32.236 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129442379-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20ae80f4 at consistency null
2022-03-22 06:19:32.236 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129442379-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:32.238 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:19:37.256 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388963089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47ae6ee3 at consistency null
2022-03-22 06:19:37.256 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [388963089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:37.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1283261109-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54773f8c at consistency null
2022-03-22 06:19:37.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1283261109-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:37.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [117728903-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@926ce14 at consistency null
2022-03-22 06:19:37.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [117728903-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:37.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [966764886-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d5b46e4 at consistency null
2022-03-22 06:19:37.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [966764886-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:37.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1451743975-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6936f2da at consistency null
2022-03-22 06:19:37.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1451743975-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:38.052 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1917522472-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4502b2f at consistency null
2022-03-22 06:19:38.052 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1917522472-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:38.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1457480376-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b3903df at consistency null
2022-03-22 06:19:38.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1457480376-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:38.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [307143724-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c521e3a at consistency null
2022-03-22 06:19:38.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [307143724-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:38.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054558816-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@622c8b48 at consistency null
2022-03-22 06:19:38.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054558816-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:39.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1342608212-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6720dd71 at consistency null
2022-03-22 06:19:39.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1342608212-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:39.784 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103566670-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14740db1 at consistency null
2022-03-22 06:19:39.784 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103566670-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:40.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [443106570-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38fb62cb at consistency null
2022-03-22 06:19:40.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [443106570-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:40.624 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:19:42.920 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:19:42.929 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:19:45.630 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349945139-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e8bd857 at consistency null
2022-03-22 06:19:45.630 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349945139-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:45.693 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873345626-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f2141a2 at consistency null
2022-03-22 06:19:45.693 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [873345626-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:45.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175965188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b8cd672 at consistency null
2022-03-22 06:19:45.748 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1175965188-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:45.863 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [369529561-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8821cf2 at consistency null
2022-03-22 06:19:45.863 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [369529561-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:45.969 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1915600949-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dd24433 at consistency null
2022-03-22 06:19:45.969 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1915600949-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:46.186 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938782920-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30603abd at consistency null
2022-03-22 06:19:46.186 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938782920-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:46.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1332428764-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@281dc5cf at consistency null
2022-03-22 06:19:46.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1332428764-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:47.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1842998450-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19214185 at consistency null
2022-03-22 06:19:47.058 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1842998450-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:47.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1911141716-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@367d7d4e at consistency null
2022-03-22 06:19:47.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1911141716-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:47.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1661544785-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7791cee at consistency null
2022-03-22 06:19:47.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1661544785-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:48.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1963618099-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@410c8273 at consistency null
2022-03-22 06:19:48.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1963618099-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:49.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469362524-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52b2debe at consistency null
2022-03-22 06:19:49.723 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469362524-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:49.729 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:19:54.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1802239471-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ff84eab at consistency null
2022-03-22 06:19:54.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1802239471-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:54.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090654834-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4029add6 at consistency null
2022-03-22 06:19:54.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090654834-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:55.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742184086-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6593b0ba at consistency null
2022-03-22 06:19:55.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1742184086-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:55.132 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [364629345-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2110d598 at consistency null
2022-03-22 06:19:55.132 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [364629345-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:55.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [554426044-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cd05a07 at consistency null
2022-03-22 06:19:55.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [554426044-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:55.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1892407661-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@611a1d56 at consistency null
2022-03-22 06:19:55.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1892407661-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:56.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [733203148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19f467af at consistency null
2022-03-22 06:19:56.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [733203148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:56.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1613285918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e6fdd1c at consistency null
2022-03-22 06:19:56.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1613285918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:57.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [522786457-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5001a8b4 at consistency null
2022-03-22 06:19:57.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [522786457-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:57.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [830973088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6140ef4f at consistency null
2022-03-22 06:19:57.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [830973088-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:58.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72875510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bb335d7 at consistency null
2022-03-22 06:19:58.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72875510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:19:58.722 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:20:03.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579448664-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bf48480 at consistency null
2022-03-22 06:20:03.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [579448664-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:03.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1967384654-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@466ddb42 at consistency null
2022-03-22 06:20:03.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1967384654-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:03.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037244130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20317e57 at consistency null
2022-03-22 06:20:03.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037244130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:04.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1122896500-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62eeea0b at consistency null
2022-03-22 06:20:04.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1122896500-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:04.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [762482206-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d01a252 at consistency null
2022-03-22 06:20:04.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [762482206-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:04.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1192821350-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@759ec57b at consistency null
2022-03-22 06:20:04.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1192821350-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:05.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373995159-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ef6de59 at consistency null
2022-03-22 06:20:05.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373995159-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:06.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1676320117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bddd93e at consistency null
2022-03-22 06:20:06.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1676320117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:07.433 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523711336-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b1ee125 at consistency null
2022-03-22 06:20:07.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523711336-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:07.445 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:20:12.458 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1440013926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18a84a20 at consistency null
2022-03-22 06:20:12.458 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1440013926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:12.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [675814771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70445b7a at consistency null
2022-03-22 06:20:12.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [675814771-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:12.574 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536755258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55757865 at consistency null
2022-03-22 06:20:12.574 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [536755258-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:12.626 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1251063824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@398ebc27 at consistency null
2022-03-22 06:20:12.626 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1251063824-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:12.696 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969242826-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@632d087c at consistency null
2022-03-22 06:20:12.696 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [969242826-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:12.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1368283396-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@316010e8 at consistency null
2022-03-22 06:20:12.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1368283396-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:12.937 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065332963-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fa8e0f1 at consistency null
2022-03-22 06:20:12.937 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065332963-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:12.949 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:20:12.954 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:20:13.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2028592676-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38b12a7 at consistency null
2022-03-22 06:20:13.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2028592676-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:13.561 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794697947-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8ee8442 at consistency null
2022-03-22 06:20:13.562 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794697947-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:14.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1226688761-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39b822e7 at consistency null
2022-03-22 06:20:14.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1226688761-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:16.041 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [187142545-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@319fe249 at consistency null
2022-03-22 06:20:16.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [187142545-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:16.043 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:20:21.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1032414767-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a5ac0f7 at consistency null
2022-03-22 06:20:21.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1032414767-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:21.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269155703-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@166a956a at consistency null
2022-03-22 06:20:21.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269155703-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:21.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938976655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e3d325a at consistency null
2022-03-22 06:20:21.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938976655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:21.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15401465-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20fe187e at consistency null
2022-03-22 06:20:21.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [15401465-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:21.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982179285-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c1f39ee at consistency null
2022-03-22 06:20:21.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982179285-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:21.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [830672163-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e1c2ec2 at consistency null
2022-03-22 06:20:21.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [830672163-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:21.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [995233089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@593468b9 at consistency null
2022-03-22 06:20:21.938 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [995233089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:22.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1676551963-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9d9a609 at consistency null
2022-03-22 06:20:22.213 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1676551963-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:22.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [950943676-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c6c3e95 at consistency null
2022-03-22 06:20:22.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [950943676-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:22.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [292407199-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53b967e4 at consistency null
2022-03-22 06:20:22.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [292407199-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:23.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [989788406-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29b2ac8a at consistency null
2022-03-22 06:20:23.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [989788406-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:24.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674940493-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d9fbe5b at consistency null
2022-03-22 06:20:24.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674940493-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:24.220 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:20:29.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1629511465-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@600abdeb at consistency null
2022-03-22 06:20:29.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1629511465-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:29.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504245677-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@447e552d at consistency null
2022-03-22 06:20:29.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504245677-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:29.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1400524754-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61534c01 at consistency null
2022-03-22 06:20:29.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1400524754-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:29.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [755823454-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a9e7e5a at consistency null
2022-03-22 06:20:29.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [755823454-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:30.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1312172225-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cff9319 at consistency null
2022-03-22 06:20:30.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1312172225-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:30.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [565839855-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42061bb7 at consistency null
2022-03-22 06:20:30.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [565839855-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:30.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [549439701-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@394e64b2 at consistency null
2022-03-22 06:20:30.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [549439701-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:31.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1793280415-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70eba16f at consistency null
2022-03-22 06:20:31.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1793280415-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:32.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [359404578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7255eb3b at consistency null
2022-03-22 06:20:32.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [359404578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:32.108 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:20:37.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1687747615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38746a37 at consistency null
2022-03-22 06:20:37.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1687747615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:37.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [977238329-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1539448f at consistency null
2022-03-22 06:20:37.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [977238329-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:37.343 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256222800-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71b8a01b at consistency null
2022-03-22 06:20:37.343 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256222800-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:37.455 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1071161386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ce99124 at consistency null
2022-03-22 06:20:37.455 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1071161386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:37.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [478403530-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37d7635c at consistency null
2022-03-22 06:20:37.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [478403530-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:37.894 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1563729874-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68154415 at consistency null
2022-03-22 06:20:37.894 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1563729874-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:38.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345966292-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c199974 at consistency null
2022-03-22 06:20:38.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [345966292-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:38.762 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1857830301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c194362 at consistency null
2022-03-22 06:20:38.762 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1857830301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:39.165 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1826874891-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30580a3e at consistency null
2022-03-22 06:20:39.166 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1826874891-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:39.570 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365964874-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@722fc5cc at consistency null
2022-03-22 06:20:39.570 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365964874-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:40.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2031432578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@626d30c4 at consistency null
2022-03-22 06:20:40.385 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2031432578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:40.387 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:20:42.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:20:42.989 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:20:45.391 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1992993256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14996db8 at consistency null
2022-03-22 06:20:45.392 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1992993256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:45.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [90884740-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64e08cef at consistency null
2022-03-22 06:20:45.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [90884740-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:45.485 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299636253-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@686d341a at consistency null
2022-03-22 06:20:45.486 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299636253-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:45.544 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090895495-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@638dc2c3 at consistency null
2022-03-22 06:20:45.545 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090895495-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:45.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1654905510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c2697a7 at consistency null
2022-03-22 06:20:45.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1654905510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:45.667 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [835584668-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c41e338 at consistency null
2022-03-22 06:20:45.667 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [835584668-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:45.780 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079219782-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d8dd14e at consistency null
2022-03-22 06:20:45.781 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1079219782-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:45.886 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [751761741-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@501b8be6 at consistency null
2022-03-22 06:20:45.886 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [751761741-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:46.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355051160-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5888a5ed at consistency null
2022-03-22 06:20:46.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355051160-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:46.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [291459838-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b03c5e7 at consistency null
2022-03-22 06:20:46.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [291459838-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:47.349 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1306658613-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38b76268 at consistency null
2022-03-22 06:20:47.349 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1306658613-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:48.962 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [373132117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34ff8a23 at consistency null
2022-03-22 06:20:48.962 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [373132117-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:48.963 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:20:54.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703836090-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29c5f69c at consistency null
2022-03-22 06:20:54.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703836090-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:54.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [237834071-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cfc22be at consistency null
2022-03-22 06:20:54.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [237834071-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:54.233 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1395354120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f5ba36b at consistency null
2022-03-22 06:20:54.233 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1395354120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:54.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [699802651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77bd9831 at consistency null
2022-03-22 06:20:54.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [699802651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:54.459 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063508798-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71caf14c at consistency null
2022-03-22 06:20:54.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063508798-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:54.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506950194-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cfec419 at consistency null
2022-03-22 06:20:54.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506950194-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:54.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1369536042-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e5c6a16 at consistency null
2022-03-22 06:20:54.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1369536042-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:54.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1227818745-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55bb0ffc at consistency null
2022-03-22 06:20:54.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1227818745-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:55.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1756663497-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73ddc496 at consistency null
2022-03-22 06:20:55.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1756663497-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:55.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550440001-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@295908f8 at consistency null
2022-03-22 06:20:55.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550440001-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:55.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2047159892-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f8cb1f at consistency null
2022-03-22 06:20:55.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2047159892-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:55.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1269626659-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@931e093 at consistency null
2022-03-22 06:20:55.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1269626659-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:56.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [119728729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14d56c57 at consistency null
2022-03-22 06:20:56.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [119728729-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:57.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599558648-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75aab8fd at consistency null
2022-03-22 06:20:57.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1599558648-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:20:57.208 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:21:02.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208904648-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37c82564 at consistency null
2022-03-22 06:21:02.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208904648-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:02.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550132362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@317b7e76 at consistency null
2022-03-22 06:21:02.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [550132362-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:02.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808066280-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e731bda at consistency null
2022-03-22 06:21:02.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808066280-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:02.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1426538886-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e3e37b9 at consistency null
2022-03-22 06:21:02.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1426538886-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:03.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1156302607-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@582bc41f at consistency null
2022-03-22 06:21:03.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1156302607-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:03.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906484498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@324989a2 at consistency null
2022-03-22 06:21:03.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [906484498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:03.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1454005350-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24f2f9fa at consistency null
2022-03-22 06:21:03.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1454005350-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:04.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961428548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b2e9122 at consistency null
2022-03-22 06:21:04.742 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961428548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:05.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596072413-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d23b964 at consistency null
2022-03-22 06:21:05.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596072413-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:05.616 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:21:10.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [735102964-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d93bfb at consistency null
2022-03-22 06:21:10.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [735102964-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:10.685 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [790493060-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30d72f53 at consistency null
2022-03-22 06:21:10.686 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [790493060-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:10.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1744388671-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fff0fea at consistency null
2022-03-22 06:21:10.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1744388671-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:10.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128201890-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@153b870e at consistency null
2022-03-22 06:21:10.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128201890-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:11.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1394298858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c1f8f4d at consistency null
2022-03-22 06:21:11.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1394298858-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:11.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [326775770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bd8537b at consistency null
2022-03-22 06:21:11.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [326775770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:12.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1044345648-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f84c9c1 at consistency null
2022-03-22 06:21:12.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1044345648-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:13.016 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:21:13.032 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:21:13.157 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [236138637-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76ce2ac8 at consistency null
2022-03-22 06:21:13.157 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [236138637-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:13.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [134109882-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@164bb6bb at consistency null
2022-03-22 06:21:13.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [134109882-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:13.987 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:21:19.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1163419004-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f917c82 at consistency null
2022-03-22 06:21:19.034 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1163419004-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:19.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [3866644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16c26670 at consistency null
2022-03-22 06:21:19.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [3866644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:19.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [398188486-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11dd960a at consistency null
2022-03-22 06:21:19.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [398188486-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:19.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1650813953-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a6814e4 at consistency null
2022-03-22 06:21:19.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1650813953-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:19.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1699835532-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e191647 at consistency null
2022-03-22 06:21:19.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1699835532-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:19.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026746770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bed1a12 at consistency null
2022-03-22 06:21:19.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1026746770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:19.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950241821-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26fdaaa3 at consistency null
2022-03-22 06:21:19.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950241821-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:19.745 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1030892543-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ca845ac at consistency null
2022-03-22 06:21:19.745 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1030892543-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:19.879 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396250579-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a15c750 at consistency null
2022-03-22 06:21:19.880 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396250579-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:20.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037189215-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@199b758a at consistency null
2022-03-22 06:21:20.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2037189215-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:20.254 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146777043-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52787ca8 at consistency null
2022-03-22 06:21:20.254 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146777043-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:20.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1408554497-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c83c66e at consistency null
2022-03-22 06:21:20.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1408554497-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:20.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1214825507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2828bbe9 at consistency null
2022-03-22 06:21:20.527 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1214825507-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:20.754 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506152579-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a5dac43 at consistency null
2022-03-22 06:21:20.754 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506152579-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:21.246 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266076103-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bd1009c at consistency null
2022-03-22 06:21:21.247 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [266076103-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:21.801 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2062181603-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49d3093e at consistency null
2022-03-22 06:21:21.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2062181603-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:22.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1949171265-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2449141e at consistency null
2022-03-22 06:21:22.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1949171265-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:22.334 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:21:27.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486976079-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23f18b60 at consistency null
2022-03-22 06:21:27.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486976079-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:27.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1307794415-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50d7ba62 at consistency null
2022-03-22 06:21:27.510 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1307794415-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:27.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1679287246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d19de66 at consistency null
2022-03-22 06:21:27.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1679287246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:27.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136196859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@421cf22e at consistency null
2022-03-22 06:21:27.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136196859-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:27.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1471111052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f7544a5 at consistency null
2022-03-22 06:21:27.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1471111052-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:27.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5528245-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48136bcf at consistency null
2022-03-22 06:21:27.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [5528245-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:28.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535561954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b5a7645 at consistency null
2022-03-22 06:21:28.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535561954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:28.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981705079-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ef27482 at consistency null
2022-03-22 06:21:28.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981705079-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:28.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402508436-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18b9163d at consistency null
2022-03-22 06:21:28.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402508436-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:29.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [895400748-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20e9a331 at consistency null
2022-03-22 06:21:29.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [895400748-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:30.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1479703717-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d6285ca at consistency null
2022-03-22 06:21:30.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1479703717-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:30.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2141054669-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cb8c81a at consistency null
2022-03-22 06:21:30.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2141054669-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:31.347 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [114109120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c603f4e at consistency null
2022-03-22 06:21:31.347 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [114109120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:31.353 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:21:36.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1183475438-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@317dffd8 at consistency null
2022-03-22 06:21:36.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1183475438-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:36.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [227995086-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@585860e8 at consistency null
2022-03-22 06:21:36.522 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [227995086-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:36.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497932607-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f2d4cad at consistency null
2022-03-22 06:21:36.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497932607-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:36.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1926851424-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1170f9af at consistency null
2022-03-22 06:21:36.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1926851424-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:36.834 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1735515843-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c47237d at consistency null
2022-03-22 06:21:36.834 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1735515843-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:37.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572054910-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@692a567c at consistency null
2022-03-22 06:21:37.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1572054910-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:37.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [782563798-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5139a212 at consistency null
2022-03-22 06:21:37.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [782563798-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:37.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [387908873-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24903f83 at consistency null
2022-03-22 06:21:37.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [387908873-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:37.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2129745647-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fb204db at consistency null
2022-03-22 06:21:37.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2129745647-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:37.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1043961368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b7cf295 at consistency null
2022-03-22 06:21:37.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1043961368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:38.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [937075264-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77c9f4da at consistency null
2022-03-22 06:21:38.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [937075264-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:38.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484971775-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5280b187 at consistency null
2022-03-22 06:21:38.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484971775-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:39.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [749780270-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d9bf3d0 at consistency null
2022-03-22 06:21:39.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [749780270-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:40.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1582000118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@517f5aac at consistency null
2022-03-22 06:21:40.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1582000118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:40.321 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:21:43.045 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:21:43.057 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:21:45.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1012991709-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69280b7d at consistency null
2022-03-22 06:21:45.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1012991709-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:45.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1918517896-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e739356 at consistency null
2022-03-22 06:21:45.363 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1918517896-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:45.392 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1537919008-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c74f23b at consistency null
2022-03-22 06:21:45.392 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1537919008-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:45.454 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1591435380-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17f57523 at consistency null
2022-03-22 06:21:45.455 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1591435380-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:45.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1782108828-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e2ff743 at consistency null
2022-03-22 06:21:45.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1782108828-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:45.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1247183697-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ce43af3 at consistency null
2022-03-22 06:21:45.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1247183697-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:45.772 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202764275-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71af67e at consistency null
2022-03-22 06:21:45.772 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202764275-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:45.891 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1122875825-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b7ac7d3 at consistency null
2022-03-22 06:21:45.891 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1122875825-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:45.999 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1852649282-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7aaeba10 at consistency null
2022-03-22 06:21:45.999 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1852649282-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:46.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [274687445-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47a02414 at consistency null
2022-03-22 06:21:46.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [274687445-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:46.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1382168844-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17d98398 at consistency null
2022-03-22 06:21:46.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1382168844-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:46.724 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2104304320-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a708aef at consistency null
2022-03-22 06:21:46.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2104304320-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:47.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2055013648-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7501c18b at consistency null
2022-03-22 06:21:47.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2055013648-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:48.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [300398151-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33c7cbbb at consistency null
2022-03-22 06:21:48.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [300398151-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:48.014 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:21:53.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1376671534-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68594b18 at consistency null
2022-03-22 06:21:53.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1376671534-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:53.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1180767373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1680930e at consistency null
2022-03-22 06:21:53.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1180767373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:53.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420221652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6df615cc at consistency null
2022-03-22 06:21:53.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1420221652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:53.756 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2019495305-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@606181da at consistency null
2022-03-22 06:21:53.756 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2019495305-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:54.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1481147019-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42c21745 at consistency null
2022-03-22 06:21:54.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1481147019-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:54.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [373502067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5048e4f0 at consistency null
2022-03-22 06:21:54.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [373502067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:54.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1952521747-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d4dd340 at consistency null
2022-03-22 06:21:54.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1952521747-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:55.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1951561299-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61cefc62 at consistency null
2022-03-22 06:21:55.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1951561299-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:55.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1746071176-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33e4b721 at consistency null
2022-03-22 06:21:55.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1746071176-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:56.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1734187857-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@241cbac9 at consistency null
2022-03-22 06:21:56.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1734187857-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:21:56.729 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:22:01.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [456163375-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4728aad9 at consistency null
2022-03-22 06:22:01.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [456163375-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:01.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2092408308-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@511363b2 at consistency null
2022-03-22 06:22:01.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2092408308-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:02.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [61753924-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47588b0e at consistency null
2022-03-22 06:22:02.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [61753924-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:02.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [144887411-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d3bbde at consistency null
2022-03-22 06:22:02.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [144887411-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:02.271 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [395052720-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4afa7bf at consistency null
2022-03-22 06:22:02.271 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [395052720-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:02.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469342088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e6b5a6b at consistency null
2022-03-22 06:22:02.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1469342088-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:02.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [647626068-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4de8b853 at consistency null
2022-03-22 06:22:02.908 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [647626068-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:03.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [945188514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@567ef77b at consistency null
2022-03-22 06:22:03.432 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [945188514-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:04.269 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1335379614-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f952054 at consistency null
2022-03-22 06:22:04.269 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1335379614-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:04.287 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:22:09.298 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005905604-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bb98fcf at consistency null
2022-03-22 06:22:09.298 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1005905604-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:09.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1391382451-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@339b82cb at consistency null
2022-03-22 06:22:09.332 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1391382451-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:09.390 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506187314-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@127f6eec at consistency null
2022-03-22 06:22:09.390 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1506187314-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:09.448 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1416071458-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14ee7b77 at consistency null
2022-03-22 06:22:09.448 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1416071458-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:09.553 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211729836-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63de49d2 at consistency null
2022-03-22 06:22:09.553 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211729836-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:09.800 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821268559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cbc02b8 at consistency null
2022-03-22 06:22:09.800 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821268559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:10.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [31738084-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@655d05cd at consistency null
2022-03-22 06:22:10.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [31738084-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:11.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1772285795-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20fefd2 at consistency null
2022-03-22 06:22:11.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1772285795-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:11.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1839337756-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@138eb698 at consistency null
2022-03-22 06:22:11.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1839337756-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:11.835 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:22:13.072 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:22:13.080 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:22:16.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1935700677-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7fcbcf93 at consistency null
2022-03-22 06:22:16.845 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1935700677-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:16.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348515621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72b5deca at consistency null
2022-03-22 06:22:16.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348515621-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:17.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776438717-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f39de31 at consistency null
2022-03-22 06:22:17.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776438717-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:17.149 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401970072-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69e58338 at consistency null
2022-03-22 06:22:17.149 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1401970072-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:17.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [588704629-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48736ec0 at consistency null
2022-03-22 06:22:17.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [588704629-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:17.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981462957-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14ecaa8a at consistency null
2022-03-22 06:22:17.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1981462957-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:18.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1683130040-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10dc83a7 at consistency null
2022-03-22 06:22:18.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1683130040-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:18.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897542967-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@430afa12 at consistency null
2022-03-22 06:22:18.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897542967-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:19.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [556164014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3243b003 at consistency null
2022-03-22 06:22:19.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [556164014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:19.422 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:22:24.511 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [38953462-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ee9569d at consistency null
2022-03-22 06:22:24.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [38953462-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:24.631 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1925533762-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a5c255c at consistency null
2022-03-22 06:22:24.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1925533762-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:24.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1009626110-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e92926f at consistency null
2022-03-22 06:22:24.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1009626110-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:24.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [772052166-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fefc16 at consistency null
2022-03-22 06:22:24.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [772052166-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:25.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1061686561-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a70f0a at consistency null
2022-03-22 06:22:25.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1061686561-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:25.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1733762506-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e0aac27 at consistency null
2022-03-22 06:22:25.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1733762506-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:25.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921970821-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64b14c63 at consistency null
2022-03-22 06:22:25.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [921970821-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:26.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1021061177-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12d902af at consistency null
2022-03-22 06:22:26.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1021061177-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:26.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1124211590-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fbb90a1 at consistency null
2022-03-22 06:22:26.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1124211590-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:27.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484763489-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24db6ae5 at consistency null
2022-03-22 06:22:27.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484763489-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:28.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [197350520-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a67e849 at consistency null
2022-03-22 06:22:28.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [197350520-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:28.206 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:22:33.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1443051737-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10f157d0 at consistency null
2022-03-22 06:22:33.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1443051737-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:33.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [93247412-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@633cf3f7 at consistency null
2022-03-22 06:22:33.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [93247412-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:33.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1047197579-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c4f2137 at consistency null
2022-03-22 06:22:33.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1047197579-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:33.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [798692522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@691fa757 at consistency null
2022-03-22 06:22:33.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [798692522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:33.623 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620124784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73ca3273 at consistency null
2022-03-22 06:22:33.624 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620124784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:33.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [324107425-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@776dc4b6 at consistency null
2022-03-22 06:22:33.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [324107425-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:33.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054031343-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@124e05f5 at consistency null
2022-03-22 06:22:33.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054031343-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:34.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229073484-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ff2df4 at consistency null
2022-03-22 06:22:34.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [229073484-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:34.341 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1685878060-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2404def2 at consistency null
2022-03-22 06:22:34.341 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1685878060-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:34.585 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2085665142-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dd5f6b0 at consistency null
2022-03-22 06:22:34.585 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2085665142-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:34.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1988855916-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f563d52 at consistency null
2022-03-22 06:22:34.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1988855916-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:35.052 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074278899-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20393bc1 at consistency null
2022-03-22 06:22:35.052 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074278899-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:35.488 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197144523-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2248827a at consistency null
2022-03-22 06:22:35.488 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197144523-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:35.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1052275551-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30ba80f5 at consistency null
2022-03-22 06:22:35.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1052275551-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:36.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [341459980-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e68971d at consistency null
2022-03-22 06:22:36.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [341459980-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:36.728 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:22:41.761 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366770397-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37664822 at consistency null
2022-03-22 06:22:41.761 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366770397-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:41.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [89677580-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51db57e6 at consistency null
2022-03-22 06:22:41.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [89677580-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:41.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2013783667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3582c800 at consistency null
2022-03-22 06:22:41.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2013783667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:42.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1044096859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f4871d2 at consistency null
2022-03-22 06:22:42.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1044096859-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:42.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213676241-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2aea1b41 at consistency null
2022-03-22 06:22:42.235 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213676241-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:42.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [263240936-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21051d9d at consistency null
2022-03-22 06:22:42.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [263240936-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:42.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1212409619-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2640375 at consistency null
2022-03-22 06:22:42.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1212409619-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:42.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [723619903-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e4e240a at consistency null
2022-03-22 06:22:42.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [723619903-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:43.101 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:22:43.121 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:22:43.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523689016-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14041a91 at consistency null
2022-03-22 06:22:43.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1523689016-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:43.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1351353974-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d641479 at consistency null
2022-03-22 06:22:43.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1351353974-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:43.767 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [127928331-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f39c30b at consistency null
2022-03-22 06:22:43.768 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [127928331-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:44.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [539476136-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20c8de92 at consistency null
2022-03-22 06:22:44.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [539476136-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:45.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1997554418-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14ac8888 at consistency null
2022-03-22 06:22:45.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1997554418-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:45.036 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:22:50.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1201624528-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@295691b8 at consistency null
2022-03-22 06:22:50.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1201624528-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:50.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081671032-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58862e02 at consistency null
2022-03-22 06:22:50.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081671032-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:50.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1581533023-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e5c47ce at consistency null
2022-03-22 06:22:50.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1581533023-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:50.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724994487-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ec46045 at consistency null
2022-03-22 06:22:50.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [724994487-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:50.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372930772-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5382e374 at consistency null
2022-03-22 06:22:50.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [372930772-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:50.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [296666198-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a9d3138 at consistency null
2022-03-22 06:22:50.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [296666198-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:50.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1831606147-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66236392 at consistency null
2022-03-22 06:22:50.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1831606147-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:51.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1917341856-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b88f15b at consistency null
2022-03-22 06:22:51.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1917341856-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:51.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938572517-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fd53a1d at consistency null
2022-03-22 06:22:51.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [938572517-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:52.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1352303673-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c12d62e at consistency null
2022-03-22 06:22:52.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1352303673-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:52.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1560740262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61d009cd at consistency null
2022-03-22 06:22:52.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1560740262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:53.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845135350-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d1e2e85 at consistency null
2022-03-22 06:22:53.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845135350-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:53.241 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:22:58.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1365307134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bfdece9 at consistency null
2022-03-22 06:22:58.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1365307134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:58.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1440682919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4154e01e at consistency null
2022-03-22 06:22:58.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1440682919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:58.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808460939-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@202b1798 at consistency null
2022-03-22 06:22:58.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [808460939-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:58.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [164415052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c1d6f1e at consistency null
2022-03-22 06:22:58.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [164415052-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:59.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603133077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58234ed0 at consistency null
2022-03-22 06:22:59.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1603133077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:59.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919069416-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e9101da at consistency null
2022-03-22 06:22:59.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1919069416-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:22:59.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1131936853-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46fe5647 at consistency null
2022-03-22 06:22:59.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1131936853-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:00.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [744926417-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@518ca419 at consistency null
2022-03-22 06:23:00.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [744926417-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:00.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497989310-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@139b3536 at consistency null
2022-03-22 06:23:00.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [497989310-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:01.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2060213263-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2532794e at consistency null
2022-03-22 06:23:01.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2060213263-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:01.409 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:23:06.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [111514199-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25528d8d at consistency null
2022-03-22 06:23:06.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [111514199-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:06.556 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [524068470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6faa2e94 at consistency null
2022-03-22 06:23:06.556 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [524068470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:06.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2001729641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59e4d226 at consistency null
2022-03-22 06:23:06.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2001729641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:06.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1870397099-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a13aa8c at consistency null
2022-03-22 06:23:06.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1870397099-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:06.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172762169-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b1c72a6 at consistency null
2022-03-22 06:23:06.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172762169-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:07.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1058271205-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15fc27df at consistency null
2022-03-22 06:23:07.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1058271205-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:07.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [932697310-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a76b2a4 at consistency null
2022-03-22 06:23:07.138 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [932697310-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:07.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [128580129-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6db622ac at consistency null
2022-03-22 06:23:07.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [128580129-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:07.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1976327115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25f7e4fe at consistency null
2022-03-22 06:23:07.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1976327115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:08.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1426128464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51c1ed40 at consistency null
2022-03-22 06:23:08.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1426128464-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:08.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555624202-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e7d964e at consistency null
2022-03-22 06:23:08.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555624202-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:08.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [336097358-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5202766f at consistency null
2022-03-22 06:23:08.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [336097358-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:09.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [51213063-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48c8a8c5 at consistency null
2022-03-22 06:23:09.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [51213063-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:09.640 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:23:13.130 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:23:13.140 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:23:14.649 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1361964872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@163df771 at consistency null
2022-03-22 06:23:14.649 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1361964872-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:14.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1250325702-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7410cc1e at consistency null
2022-03-22 06:23:14.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1250325702-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:14.762 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [500951000-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35607dfd at consistency null
2022-03-22 06:23:14.762 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [500951000-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:14.856 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1973828143-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38636514 at consistency null
2022-03-22 06:23:14.856 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1973828143-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:14.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1271913596-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@419c9b16 at consistency null
2022-03-22 06:23:14.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1271913596-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:14.966 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [840343943-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@331edf07 at consistency null
2022-03-22 06:23:14.966 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [840343943-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:15.059 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1882285370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@527d30c at consistency null
2022-03-22 06:23:15.059 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1882285370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:15.184 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [880954961-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@264f5d79 at consistency null
2022-03-22 06:23:15.184 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [880954961-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:15.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18569345-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ae368d at consistency null
2022-03-22 06:23:15.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18569345-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:15.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109497276-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@460d2718 at consistency null
2022-03-22 06:23:15.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [109497276-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:15.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2112186083-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d893366 at consistency null
2022-03-22 06:23:15.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2112186083-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:15.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1530093512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21711d3b at consistency null
2022-03-22 06:23:15.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1530093512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:15.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12049279-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53994dc6 at consistency null
2022-03-22 06:23:15.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12049279-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:15.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [811346942-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c03954a at consistency null
2022-03-22 06:23:15.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [811346942-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:16.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1725464936-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48a59baf at consistency null
2022-03-22 06:23:16.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1725464936-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:16.664 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [668229476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6114cf45 at consistency null
2022-03-22 06:23:16.664 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [668229476-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:17.087 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [419824834-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@787a49dc at consistency null
2022-03-22 06:23:17.087 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [419824834-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:17.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1221910471-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18333a5c at consistency null
2022-03-22 06:23:17.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1221910471-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:17.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1789252724-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c2fa216 at consistency null
2022-03-22 06:23:17.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1789252724-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:17.922 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:23:23.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871677755-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b862431 at consistency null
2022-03-22 06:23:23.128 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1871677755-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:23.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1183232028-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1198ae34 at consistency null
2022-03-22 06:23:23.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1183232028-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:23.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683832812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10f88781 at consistency null
2022-03-22 06:23:23.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683832812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:23.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820328692-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11c6a89a at consistency null
2022-03-22 06:23:23.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1820328692-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:23.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136315918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1beba13f at consistency null
2022-03-22 06:23:23.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136315918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:24.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [203473320-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f565578 at consistency null
2022-03-22 06:23:24.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [203473320-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:24.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [930527445-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b46079c at consistency null
2022-03-22 06:23:24.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [930527445-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:24.867 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1536002925-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fe0fe16 at consistency null
2022-03-22 06:23:24.870 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1536002925-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:25.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147586554-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79c4428c at consistency null
2022-03-22 06:23:25.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147586554-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:25.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49037514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4443559 at consistency null
2022-03-22 06:23:25.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [49037514-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:26.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [14610514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f15b8aa at consistency null
2022-03-22 06:23:26.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [14610514-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:26.319 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:23:31.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [926926279-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@173df310 at consistency null
2022-03-22 06:23:31.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [926926279-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:31.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1624034039-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c361359 at consistency null
2022-03-22 06:23:31.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1624034039-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:31.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12987444-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@697d097b at consistency null
2022-03-22 06:23:31.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [12987444-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:31.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2003636254-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6baf2c16 at consistency null
2022-03-22 06:23:31.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2003636254-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:31.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1052107465-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1372e700 at consistency null
2022-03-22 06:23:31.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1052107465-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:32.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [950537370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f2282ec at consistency null
2022-03-22 06:23:32.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [950537370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:32.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155070519-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e1568a2 at consistency null
2022-03-22 06:23:32.428 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155070519-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:32.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [374090990-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c4836ea at consistency null
2022-03-22 06:23:32.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [374090990-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:33.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [125708100-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78068484 at consistency null
2022-03-22 06:23:33.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [125708100-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:34.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2135566317-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3484c8a at consistency null
2022-03-22 06:23:34.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2135566317-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:34.041 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:23:39.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [126689211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46dd7439 at consistency null
2022-03-22 06:23:39.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [126689211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:39.105 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1892772088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b9fe56e at consistency null
2022-03-22 06:23:39.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1892772088-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:39.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [115549601-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37717c3d at consistency null
2022-03-22 06:23:39.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [115549601-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:39.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683155423-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@769c4a45 at consistency null
2022-03-22 06:23:39.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [683155423-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:39.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202171316-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b4d9413 at consistency null
2022-03-22 06:23:39.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202171316-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:39.640 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1019910928-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ffdbe81 at consistency null
2022-03-22 06:23:39.640 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1019910928-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:39.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589000453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63d1b1d7 at consistency null
2022-03-22 06:23:39.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589000453-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:40.088 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1270209118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@770b018a at consistency null
2022-03-22 06:23:40.089 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1270209118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:40.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [822416529-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40d1fe74 at consistency null
2022-03-22 06:23:40.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [822416529-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:40.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1509891002-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@726a5959 at consistency null
2022-03-22 06:23:40.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1509891002-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:41.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568599751-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c5b01ca at consistency null
2022-03-22 06:23:41.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [568599751-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:41.538 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:23:43.158 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:23:43.180 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:23:46.563 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1147567503-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33f3549a at consistency null
2022-03-22 06:23:46.563 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1147567503-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:46.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1574848526-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c9e3ece at consistency null
2022-03-22 06:23:46.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1574848526-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:46.753 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2094901956-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c1e1095 at consistency null
2022-03-22 06:23:46.753 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2094901956-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:46.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2090826628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@188f0f8d at consistency null
2022-03-22 06:23:46.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2090826628-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:46.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [191032221-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31f1df82 at consistency null
2022-03-22 06:23:46.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [191032221-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609443015-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c94ff3c at consistency null
2022-03-22 06:23:47.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [609443015-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.073 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [893999363-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1022da83 at consistency null
2022-03-22 06:23:47.073 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [893999363-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839735157-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70b14c6d at consistency null
2022-03-22 06:23:47.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [839735157-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.262 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [783252650-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f038036 at consistency null
2022-03-22 06:23:47.263 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [783252650-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.324 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [279226588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54f5ebf5 at consistency null
2022-03-22 06:23:47.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [279226588-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.398 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1876589194-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5205d52a at consistency null
2022-03-22 06:23:47.398 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1876589194-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [934559483-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@282d2923 at consistency null
2022-03-22 06:23:47.452 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [934559483-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1655393931-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bfc0e20 at consistency null
2022-03-22 06:23:47.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1655393931-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1476196426-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ade8e9 at consistency null
2022-03-22 06:23:47.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1476196426-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1800328725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c7dd153 at consistency null
2022-03-22 06:23:47.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1800328725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2098949729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3dd89bce at consistency null
2022-03-22 06:23:47.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2098949729-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:47.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1165801567-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26ce1f5b at consistency null
2022-03-22 06:23:47.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1165801567-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:48.034 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1978803730-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49cb41cb at consistency null
2022-03-22 06:23:48.034 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1978803730-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:48.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754380487-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b2ef6e8 at consistency null
2022-03-22 06:23:48.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1754380487-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:48.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1323285458-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22d110a8 at consistency null
2022-03-22 06:23:48.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1323285458-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:48.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1858161686-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67e5253a at consistency null
2022-03-22 06:23:48.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1858161686-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:49.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [174531946-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b500698 at consistency null
2022-03-22 06:23:49.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [174531946-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:49.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1876288353-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@612e0b0a at consistency null
2022-03-22 06:23:49.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1876288353-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:50.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550304354-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a4d08c6 at consistency null
2022-03-22 06:23:50.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550304354-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:50.520 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:23:55.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941785604-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44cf2040 at consistency null
2022-03-22 06:23:55.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941785604-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:55.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199080991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f609a87 at consistency null
2022-03-22 06:23:55.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199080991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:55.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1272154350-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e07b63a at consistency null
2022-03-22 06:23:55.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1272154350-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:56.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695537954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cee8604 at consistency null
2022-03-22 06:23:56.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695537954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:56.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [90936490-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@419e74f9 at consistency null
2022-03-22 06:23:56.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [90936490-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:56.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1664154972-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a216aef at consistency null
2022-03-22 06:23:56.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1664154972-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:57.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879988012-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69b4b3f7 at consistency null
2022-03-22 06:23:57.643 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [879988012-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:59.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [898528897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61e2c663 at consistency null
2022-03-22 06:23:59.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [898528897-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:23:59.338 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:24:04.441 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821997662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53695f1a at consistency null
2022-03-22 06:24:04.441 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1821997662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:04.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136814732-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@125e318a at consistency null
2022-03-22 06:24:04.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136814732-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:04.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [546653769-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5eb9133d at consistency null
2022-03-22 06:24:04.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [546653769-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:04.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1726547620-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@299f0757 at consistency null
2022-03-22 06:24:04.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1726547620-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:04.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099287469-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30171cf1 at consistency null
2022-03-22 06:24:04.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099287469-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:05.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1947410641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5326a918 at consistency null
2022-03-22 06:24:05.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1947410641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:05.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2118031124-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@451068ba at consistency null
2022-03-22 06:24:05.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2118031124-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:05.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355571793-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14646af0 at consistency null
2022-03-22 06:24:05.301 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355571793-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:05.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065030470-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c1c0c19 at consistency null
2022-03-22 06:24:05.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1065030470-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:05.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1375971792-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a9418eb at consistency null
2022-03-22 06:24:05.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1375971792-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:06.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1477382835-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c87fabb at consistency null
2022-03-22 06:24:06.267 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1477382835-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:07.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809988873-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b832431 at consistency null
2022-03-22 06:24:07.120 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809988873-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:08.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [216326672-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77839d9c at consistency null
2022-03-22 06:24:08.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [216326672-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:08.007 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:24:13.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1139181429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d0a80c7 at consistency null
2022-03-22 06:24:13.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1139181429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:13.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1675403056-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e5be5ff at consistency null
2022-03-22 06:24:13.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1675403056-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:13.189 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:24:13.206 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:24:13.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [422378551-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cebfea9 at consistency null
2022-03-22 06:24:13.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [422378551-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:13.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [854813024-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57effbd5 at consistency null
2022-03-22 06:24:13.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [854813024-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:13.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1163500900-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e1a5557 at consistency null
2022-03-22 06:24:13.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1163500900-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:13.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1115402208-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f1f3976 at consistency null
2022-03-22 06:24:13.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1115402208-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:13.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1019752156-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ce29a8a at consistency null
2022-03-22 06:24:13.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1019752156-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:14.142 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366606412-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@147bef90 at consistency null
2022-03-22 06:24:14.143 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366606412-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:14.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484631854-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6837b0a5 at consistency null
2022-03-22 06:24:14.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [484631854-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:14.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [58281469-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e45f05c at consistency null
2022-03-22 06:24:14.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [58281469-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:15.066 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2126130560-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66db05d4 at consistency null
2022-03-22 06:24:15.066 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2126130560-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:15.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [620474755-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d1771f0 at consistency null
2022-03-22 06:24:15.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [620474755-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:16.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [792460199-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78f05e6f at consistency null
2022-03-22 06:24:16.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [792460199-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:16.322 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:24:21.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1717853464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1956863d at consistency null
2022-03-22 06:24:21.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1717853464-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:21.759 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235073657-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@279dd566 at consistency null
2022-03-22 06:24:21.759 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235073657-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:21.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [814972256-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3193b894 at consistency null
2022-03-22 06:24:21.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [814972256-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:22.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074089494-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b6a8bef at consistency null
2022-03-22 06:24:22.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074089494-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:22.357 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72173459-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66a08bd1 at consistency null
2022-03-22 06:24:22.358 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72173459-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:22.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [539458321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f5f4486 at consistency null
2022-03-22 06:24:22.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [539458321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:23.248 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [192182792-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3240ee1 at consistency null
2022-03-22 06:24:23.249 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [192182792-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:23.749 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1642852494-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e1236ca at consistency null
2022-03-22 06:24:23.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1642852494-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:24.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [267666175-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bbc26fa at consistency null
2022-03-22 06:24:24.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [267666175-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:25.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [336617443-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8cc766a at consistency null
2022-03-22 06:24:25.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [336617443-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:25.122 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:24:30.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1008522067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62ec3dea at consistency null
2022-03-22 06:24:30.222 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1008522067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:30.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47004638-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d27f31b at consistency null
2022-03-22 06:24:30.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [47004638-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:30.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [788755138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47668fb3 at consistency null
2022-03-22 06:24:30.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [788755138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:30.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [972738438-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c72ce13 at consistency null
2022-03-22 06:24:30.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [972738438-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:30.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1671828897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f72b51a at consistency null
2022-03-22 06:24:30.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1671828897-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:30.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [140427712-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dff34c4 at consistency null
2022-03-22 06:24:30.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [140427712-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:30.957 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1014966123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e172bdf at consistency null
2022-03-22 06:24:30.957 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1014966123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:31.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1402362269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@542f7258 at consistency null
2022-03-22 06:24:31.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1402362269-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:31.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [13518163-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ecfc7a2 at consistency null
2022-03-22 06:24:31.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [13518163-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:32.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128663627-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@af287ab at consistency null
2022-03-22 06:24:32.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128663627-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:33.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [996270211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d6c1cf9 at consistency null
2022-03-22 06:24:33.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [996270211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:33.218 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:24:38.260 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524243651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72fe08be at consistency null
2022-03-22 06:24:38.260 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524243651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:38.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [676081174-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@621c3394 at consistency null
2022-03-22 06:24:38.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [676081174-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:38.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349969368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c4d3fa3 at consistency null
2022-03-22 06:24:38.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349969368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:38.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1688695469-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@417b6f07 at consistency null
2022-03-22 06:24:38.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1688695469-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:38.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1846630142-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43c2346e at consistency null
2022-03-22 06:24:38.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1846630142-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:38.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [660928948-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40413f11 at consistency null
2022-03-22 06:24:38.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [660928948-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:38.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1088127141-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55e4b09d at consistency null
2022-03-22 06:24:38.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1088127141-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:38.851 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523677705-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ffdfe71 at consistency null
2022-03-22 06:24:38.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523677705-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:39.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2071245751-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d4ec2a4 at consistency null
2022-03-22 06:24:39.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2071245751-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:39.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2078104628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@543ecfab at consistency null
2022-03-22 06:24:39.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2078104628-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:39.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1730098498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24aeede9 at consistency null
2022-03-22 06:24:39.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1730098498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:40.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1292068688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@480917b at consistency null
2022-03-22 06:24:40.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1292068688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:40.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [330599500-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fb028ad at consistency null
2022-03-22 06:24:40.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [330599500-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:41.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43601650-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dd1f167 at consistency null
2022-03-22 06:24:41.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43601650-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:41.824 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:24:43.224 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:24:43.239 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:24:46.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [473095878-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e439c91 at consistency null
2022-03-22 06:24:46.840 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [473095878-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:46.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1247609847-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e48dd6b at consistency null
2022-03-22 06:24:46.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1247609847-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:47.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2101096476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70536b4f at consistency null
2022-03-22 06:24:47.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2101096476-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:47.249 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890904471-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32cbe6eb at consistency null
2022-03-22 06:24:47.249 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [890904471-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:47.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1062482716-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@306e5337 at consistency null
2022-03-22 06:24:47.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1062482716-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:47.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [498392506-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@689573d4 at consistency null
2022-03-22 06:24:47.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [498392506-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:48.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18133696-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f7649a at consistency null
2022-03-22 06:24:48.129 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [18133696-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:48.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1679291346-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e6626b1 at consistency null
2022-03-22 06:24:48.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1679291346-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:49.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2059792543-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cd337a1 at consistency null
2022-03-22 06:24:49.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2059792543-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:49.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1093122928-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f1bea3c at consistency null
2022-03-22 06:24:49.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1093122928-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:49.924 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:24:55.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1694597926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@577090f1 at consistency null
2022-03-22 06:24:55.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1694597926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:55.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1054045495-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@87c6700 at consistency null
2022-03-22 06:24:55.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1054045495-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:55.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1799826298-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3da73791 at consistency null
2022-03-22 06:24:55.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1799826298-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:55.553 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [398975321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ddd6013 at consistency null
2022-03-22 06:24:55.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [398975321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:55.869 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117486644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ef99e41 at consistency null
2022-03-22 06:24:55.869 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117486644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:56.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1533665288-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3556b6f9 at consistency null
2022-03-22 06:24:56.334 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1533665288-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:56.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [153558244-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31c9047d at consistency null
2022-03-22 06:24:56.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [153558244-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:57.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [566421936-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@577227af at consistency null
2022-03-22 06:24:57.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [566421936-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:24:57.626 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:25:02.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845060180-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45ebb096 at consistency null
2022-03-22 06:25:02.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1845060180-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:03.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2030351939-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@450a37ac at consistency null
2022-03-22 06:25:03.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2030351939-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:03.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215645115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c99299c at consistency null
2022-03-22 06:25:03.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215645115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:03.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2118957676-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a102eff at consistency null
2022-03-22 06:25:03.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2118957676-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:03.450 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484889139-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62e74e2b at consistency null
2022-03-22 06:25:03.450 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1484889139-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:03.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74158030-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76c4b28c at consistency null
2022-03-22 06:25:03.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74158030-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:03.627 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1716603493-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@364a001 at consistency null
2022-03-22 06:25:03.627 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1716603493-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:03.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2111111961-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78f81144 at consistency null
2022-03-22 06:25:03.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2111111961-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:04.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [702638228-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20b68e66 at consistency null
2022-03-22 06:25:04.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [702638228-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:04.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1907132313-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73b5082e at consistency null
2022-03-22 06:25:04.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1907132313-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:04.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [490924459-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55f7caf1 at consistency null
2022-03-22 06:25:04.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [490924459-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:04.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213922513-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6dbd833a at consistency null
2022-03-22 06:25:04.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1213922513-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:04.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1792907398-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79ecf555 at consistency null
2022-03-22 06:25:04.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1792907398-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:05.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136492555-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22c3663e at consistency null
2022-03-22 06:25:05.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2136492555-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:05.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1977482234-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@498880d5 at consistency null
2022-03-22 06:25:05.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1977482234-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:05.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1795042788-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48ad6e8d at consistency null
2022-03-22 06:25:05.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1795042788-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:06.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1818333589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56757c90 at consistency null
2022-03-22 06:25:06.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1818333589-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:06.250 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:25:06.790 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 06:25:06.808 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-48, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 06:25:06.808 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-48, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 06:25:06.809 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-48, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 06:25:06.817 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-48, inFlight=0, closed=true] closing connection
2022-03-22 06:25:06.819 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 06:25:06.819 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 600000 ms
2022-03-22 06:25:06.825 DEBUG 10 --- [uster-worker-87] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 06:25:11.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1372886528-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fbe02f6 at consistency null
2022-03-22 06:25:11.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1372886528-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:11.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1271707115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62bdb9fc at consistency null
2022-03-22 06:25:11.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1271707115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:11.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355161723-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41476a20 at consistency null
2022-03-22 06:25:11.386 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355161723-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:11.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [818671311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cab0f64 at consistency null
2022-03-22 06:25:11.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [818671311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:11.590 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1163532864-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@231d13b5 at consistency null
2022-03-22 06:25:11.590 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1163532864-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:11.650 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1796705517-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ce03c73 at consistency null
2022-03-22 06:25:11.650 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1796705517-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:11.775 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171158540-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@568c55a5 at consistency null
2022-03-22 06:25:11.775 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [171158540-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:11.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1709278739-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3238b03f at consistency null
2022-03-22 06:25:11.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1709278739-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:12.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887181045-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79739d76 at consistency null
2022-03-22 06:25:12.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887181045-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:12.540 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [660308723-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54dd358f at consistency null
2022-03-22 06:25:12.541 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [660308723-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:13.240 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:25:13.255 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:25:13.360 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [815733208-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@604873cc at consistency null
2022-03-22 06:25:13.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [815733208-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:15.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495777478-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17868eb2 at consistency null
2022-03-22 06:25:15.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495777478-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:15.011 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:25:20.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1225576615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72fb3919 at consistency null
2022-03-22 06:25:20.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1225576615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:20.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851152751-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fb71b84 at consistency null
2022-03-22 06:25:20.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851152751-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:20.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2057441593-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c42c7ad at consistency null
2022-03-22 06:25:20.410 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2057441593-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:20.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493150043-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f923789 at consistency null
2022-03-22 06:25:20.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [493150043-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:20.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851623352-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@648aec93 at consistency null
2022-03-22 06:25:20.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [851623352-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:21.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550617720-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1396a0bd at consistency null
2022-03-22 06:25:21.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550617720-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:21.662 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917337365-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@691362cd at consistency null
2022-03-22 06:25:21.662 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917337365-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:22.141 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [218692252-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37633e19 at consistency null
2022-03-22 06:25:22.141 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [218692252-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:23.060 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [704690680-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43ea212 at consistency null
2022-03-22 06:25:23.061 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [704690680-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:23.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [305655692-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@228b6960 at consistency null
2022-03-22 06:25:23.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [305655692-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:23.927 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:25:29.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341072642-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@507c72f6 at consistency null
2022-03-22 06:25:29.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1341072642-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:29.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [100264569-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ec11d16 at consistency null
2022-03-22 06:25:29.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [100264569-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:29.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [714689607-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f9e0eee at consistency null
2022-03-22 06:25:29.320 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [714689607-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:29.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [134188311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76a7a614 at consistency null
2022-03-22 06:25:29.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [134188311-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:29.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [823033823-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45ff91f4 at consistency null
2022-03-22 06:25:29.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [823033823-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:30.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079256997-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33a01df0 at consistency null
2022-03-22 06:25:30.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2079256997-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:30.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [469070315-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cb68c14 at consistency null
2022-03-22 06:25:30.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [469070315-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:30.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620689079-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ea0490 at consistency null
2022-03-22 06:25:30.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620689079-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:30.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [573291516-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e2345a8 at consistency null
2022-03-22 06:25:30.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [573291516-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:30.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776209861-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d5f8001 at consistency null
2022-03-22 06:25:30.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776209861-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:31.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504255439-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75f16b92 at consistency null
2022-03-22 06:25:31.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [504255439-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:31.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [809284801-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22683f38 at consistency null
2022-03-22 06:25:31.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [809284801-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:32.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1337396577-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c220d8 at consistency null
2022-03-22 06:25:32.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1337396577-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:32.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1104368511-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77ecf789 at consistency null
2022-03-22 06:25:32.529 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1104368511-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:32.532 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:25:37.547 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [758116513-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@93faf1d at consistency null
2022-03-22 06:25:37.547 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [758116513-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:37.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027996025-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a4dc0fd at consistency null
2022-03-22 06:25:37.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2027996025-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:37.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256574721-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d9ef98c at consistency null
2022-03-22 06:25:37.750 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256574721-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:37.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415116604-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6dde0909 at consistency null
2022-03-22 06:25:37.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415116604-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:38.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [102809582-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@196d14e8 at consistency null
2022-03-22 06:25:38.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [102809582-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:38.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2015711064-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1da37510 at consistency null
2022-03-22 06:25:38.345 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2015711064-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:38.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [798047824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17fbadb1 at consistency null
2022-03-22 06:25:38.554 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [798047824-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:39.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013173641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a706806 at consistency null
2022-03-22 06:25:39.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013173641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:39.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1069537828-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bcbe7f4 at consistency null
2022-03-22 06:25:39.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1069537828-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:40.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1831996676-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d460668 at consistency null
2022-03-22 06:25:40.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1831996676-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:40.583 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 06:25:40.595 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-72, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 06:25:40.595 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-72, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 06:25:40.595 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-72, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 06:25:40.596 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-72, inFlight=0, closed=true] closing connection
2022-03-22 06:25:40.597 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 06:25:40.597 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 600000 ms
2022-03-22 06:25:40.598 DEBUG 10 --- [uster-worker-88] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 06:25:41.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1274531148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@379aba54 at consistency null
2022-03-22 06:25:41.119 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1274531148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:41.121 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:25:43.269 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:25:43.275 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:25:46.158 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1775914558-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ea3d4f9 at consistency null
2022-03-22 06:25:46.158 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1775914558-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:46.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [678811611-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1af3609e at consistency null
2022-03-22 06:25:46.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [678811611-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:46.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1431483900-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57cb40ee at consistency null
2022-03-22 06:25:46.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1431483900-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:46.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540712887-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35610859 at consistency null
2022-03-22 06:25:46.466 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540712887-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:46.678 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [455773744-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15451610 at consistency null
2022-03-22 06:25:46.678 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [455773744-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:47.100 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [501700993-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14b373b6 at consistency null
2022-03-22 06:25:47.100 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [501700993-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:47.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1566237332-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19f28dfe at consistency null
2022-03-22 06:25:47.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1566237332-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:49.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [694753711-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24ec7d59 at consistency null
2022-03-22 06:25:49.622 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [694753711-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:49.627 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:25:54.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727875516-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19240a42 at consistency null
2022-03-22 06:25:54.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1727875516-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:54.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1097936819-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1685f413 at consistency null
2022-03-22 06:25:54.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1097936819-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:54.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261775691-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b151640 at consistency null
2022-03-22 06:25:54.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261775691-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:55.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720431075-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cd4a6ce at consistency null
2022-03-22 06:25:55.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720431075-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:55.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [642579368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@651bd10c at consistency null
2022-03-22 06:25:55.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [642579368-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:55.260 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1854424165-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c0794d0 at consistency null
2022-03-22 06:25:55.260 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1854424165-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:55.375 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [201672076-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f132e06 at consistency null
2022-03-22 06:25:55.376 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [201672076-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:55.583 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [560428048-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@168807fc at consistency null
2022-03-22 06:25:55.583 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [560428048-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:56.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1622624815-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2da03790 at consistency null
2022-03-22 06:25:56.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1622624815-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:56.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286639920-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d6187e0 at consistency null
2022-03-22 06:25:56.917 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286639920-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:58.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1790940175-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b74f354 at consistency null
2022-03-22 06:25:58.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1790940175-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:25:58.633 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:26:03.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893903036-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d15a421 at consistency null
2022-03-22 06:26:03.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1893903036-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:03.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [710828221-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@412512a4 at consistency null
2022-03-22 06:26:03.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [710828221-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:03.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1655659991-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ad3e7 at consistency null
2022-03-22 06:26:03.835 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1655659991-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:03.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1948688993-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c487ee7 at consistency null
2022-03-22 06:26:03.941 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1948688993-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:04.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [292831507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c3ce5e3 at consistency null
2022-03-22 06:26:04.126 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [292831507-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:04.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215729672-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35656cf2 at consistency null
2022-03-22 06:26:04.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1215729672-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:04.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1345273140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63963452 at consistency null
2022-03-22 06:26:04.532 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1345273140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:05.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299608314-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c76d181 at consistency null
2022-03-22 06:26:05.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299608314-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:05.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [571744011-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7de2811 at consistency null
2022-03-22 06:26:05.838 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [571744011-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:07.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579040145-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1942d6e4 at consistency null
2022-03-22 06:26:07.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579040145-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:07.618 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:26:12.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012315131-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b24553a at consistency null
2022-03-22 06:26:12.734 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2012315131-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:12.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [992053127-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41d46f06 at consistency null
2022-03-22 06:26:12.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [992053127-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:12.940 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961493935-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37bc498d at consistency null
2022-03-22 06:26:12.940 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961493935-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:13.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [378698658-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38c64c68 at consistency null
2022-03-22 06:26:13.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [378698658-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:13.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794163246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51a253cf at consistency null
2022-03-22 06:26:13.134 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794163246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:13.297 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:26:13.308 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:26:13.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1566207304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@336caaa2 at consistency null
2022-03-22 06:26:13.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1566207304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:13.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [495494927-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66497d63 at consistency null
2022-03-22 06:26:13.521 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [495494927-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:13.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407015543-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c66c8ce at consistency null
2022-03-22 06:26:13.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407015543-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:14.798 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1951383779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f19d2a6 at consistency null
2022-03-22 06:26:14.798 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1951383779-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:15.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054275855-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7976545d at consistency null
2022-03-22 06:26:15.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2054275855-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:15.617 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:26:20.635 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [572109713-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54799c75 at consistency null
2022-03-22 06:26:20.635 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [572109713-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:20.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [785669773-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e031e8c at consistency null
2022-03-22 06:26:20.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [785669773-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:20.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [605473644-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f6e6108 at consistency null
2022-03-22 06:26:20.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [605473644-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:20.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1342901594-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24710404 at consistency null
2022-03-22 06:26:20.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1342901594-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:21.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063365105-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cf3becb at consistency null
2022-03-22 06:26:21.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063365105-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:21.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [869120166-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a9220a7 at consistency null
2022-03-22 06:26:21.071 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [869120166-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:21.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [563234840-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36e862dc at consistency null
2022-03-22 06:26:21.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [563234840-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:21.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695984301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a951121 at consistency null
2022-03-22 06:26:21.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [695984301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:21.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [459795026-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bb1f15e at consistency null
2022-03-22 06:26:21.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [459795026-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:21.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1169254672-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9d7997 at consistency null
2022-03-22 06:26:21.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1169254672-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:21.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1984965430-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@751615da at consistency null
2022-03-22 06:26:21.808 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1984965430-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:22.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [14205913-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47b80e40 at consistency null
2022-03-22 06:26:22.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [14205913-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:22.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617188586-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73373ce8 at consistency null
2022-03-22 06:26:22.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [617188586-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:22.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1786681482-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@180db2d8 at consistency null
2022-03-22 06:26:22.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1786681482-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:22.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486541160-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5df0dcc6 at consistency null
2022-03-22 06:26:22.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486541160-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:22.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1526778138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5abd0336 at consistency null
2022-03-22 06:26:22.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1526778138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:23.159 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [868655180-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1027a4f5 at consistency null
2022-03-22 06:26:23.159 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [868655180-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:23.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769836150-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65494f5b at consistency null
2022-03-22 06:26:23.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [769836150-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:24.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [818200037-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e04a9d8 at consistency null
2022-03-22 06:26:24.523 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [818200037-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:24.526 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:26:29.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1834085118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c12a99d at consistency null
2022-03-22 06:26:29.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1834085118-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:29.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1796469720-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@213fa4e6 at consistency null
2022-03-22 06:26:29.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1796469720-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:29.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1357466552-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ea3040d at consistency null
2022-03-22 06:26:29.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1357466552-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:29.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1171783637-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15723668 at consistency null
2022-03-22 06:26:29.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1171783637-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:30.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [666102667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cb52b1a at consistency null
2022-03-22 06:26:30.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [666102667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:30.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1405768086-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46645391 at consistency null
2022-03-22 06:26:30.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1405768086-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:30.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1231320284-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4221fc43 at consistency null
2022-03-22 06:26:30.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1231320284-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:30.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [867636082-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67e9e2e6 at consistency null
2022-03-22 06:26:30.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [867636082-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:30.454 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199248188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5eee5ab7 at consistency null
2022-03-22 06:26:30.454 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199248188-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:30.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [170393399-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@136076cd at consistency null
2022-03-22 06:26:30.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [170393399-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:30.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611638843-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bb65895 at consistency null
2022-03-22 06:26:30.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611638843-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:31.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806179275-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1218ae36 at consistency null
2022-03-22 06:26:31.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1806179275-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:31.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1653085739-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e0f7520 at consistency null
2022-03-22 06:26:31.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1653085739-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:31.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1466143498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ec9ba32 at consistency null
2022-03-22 06:26:31.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1466143498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:32.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1253095341-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27323d8e at consistency null
2022-03-22 06:26:32.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1253095341-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:32.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1642957683-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@708c5e40 at consistency null
2022-03-22 06:26:32.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1642957683-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:32.943 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:26:38.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1131868203-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@356f8304 at consistency null
2022-03-22 06:26:38.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1131868203-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:38.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [138858636-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5614b7b1 at consistency null
2022-03-22 06:26:38.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [138858636-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:38.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1199834102-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60fc2eb0 at consistency null
2022-03-22 06:26:38.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1199834102-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:38.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1245691494-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@729002b1 at consistency null
2022-03-22 06:26:38.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1245691494-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:38.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1052571810-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3baed844 at consistency null
2022-03-22 06:26:38.461 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1052571810-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:38.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [4234569-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48f449bf at consistency null
2022-03-22 06:26:38.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [4234569-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:39.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1732027652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49fc77bf at consistency null
2022-03-22 06:26:39.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1732027652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:40.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355279115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b8b71bb at consistency null
2022-03-22 06:26:40.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355279115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:41.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [166162521-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76aaf276 at consistency null
2022-03-22 06:26:41.733 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [166162521-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:41.734 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:26:43.309 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:26:43.330 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:26:46.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [231280377-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c79e4ad at consistency null
2022-03-22 06:26:46.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [231280377-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:46.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257920705-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36923912 at consistency null
2022-03-22 06:26:46.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [257920705-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:46.851 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1622123410-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@567490d2 at consistency null
2022-03-22 06:26:46.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1622123410-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:46.922 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1392013926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77ecf128 at consistency null
2022-03-22 06:26:46.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1392013926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:47.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394461190-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24e32ff4 at consistency null
2022-03-22 06:26:47.002 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [394461190-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:47.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1683602865-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31ed8d3 at consistency null
2022-03-22 06:26:47.114 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1683602865-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:47.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046243227-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61ebf0c8 at consistency null
2022-03-22 06:26:47.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046243227-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:47.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1745688449-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c4aff8e at consistency null
2022-03-22 06:26:47.328 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1745688449-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:47.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728166326-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29e77c94 at consistency null
2022-03-22 06:26:47.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728166326-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:47.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1310471378-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29aa0ee0 at consistency null
2022-03-22 06:26:47.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1310471378-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:48.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245874531-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1779c6ad at consistency null
2022-03-22 06:26:48.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245874531-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:48.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [244715561-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c44fd43 at consistency null
2022-03-22 06:26:48.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [244715561-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:49.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680917572-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ed53183 at consistency null
2022-03-22 06:26:49.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1680917572-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:50.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1269155508-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77f1e3b7 at consistency null
2022-03-22 06:26:50.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1269155508-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:50.320 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:26:55.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026513412-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e641f4a at consistency null
2022-03-22 06:26:55.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026513412-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:55.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1189311596-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15715120 at consistency null
2022-03-22 06:26:55.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1189311596-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:55.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613653008-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@328f3324 at consistency null
2022-03-22 06:26:55.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [613653008-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:55.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1105623120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a1b8299 at consistency null
2022-03-22 06:26:55.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1105623120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:56.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1755684460-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@448b5adf at consistency null
2022-03-22 06:26:56.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1755684460-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:56.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [892917504-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b6cfb3f at consistency null
2022-03-22 06:26:56.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [892917504-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:56.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [222529709-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1031f565 at consistency null
2022-03-22 06:26:56.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [222529709-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:56.401 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2132104621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a8b2b0 at consistency null
2022-03-22 06:26:56.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2132104621-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:56.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998765277-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18eb5d18 at consistency null
2022-03-22 06:26:56.538 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998765277-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:56.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223996026-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2462f839 at consistency null
2022-03-22 06:26:56.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1223996026-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:56.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1508072745-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b67ab0d at consistency null
2022-03-22 06:26:56.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1508072745-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:57.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [210686853-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7899a3e3 at consistency null
2022-03-22 06:26:57.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [210686853-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:57.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136038867-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a582275 at consistency null
2022-03-22 06:26:57.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136038867-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:57.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1704054216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@948b637 at consistency null
2022-03-22 06:26:57.715 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1704054216-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:58.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [651742790-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f715bd0 at consistency null
2022-03-22 06:26:58.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [651742790-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:58.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1546662883-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ac27766 at consistency null
2022-03-22 06:26:58.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1546662883-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:59.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179518695-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d475ed6 at consistency null
2022-03-22 06:26:59.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179518695-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:26:59.018 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:27:04.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1629970420-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8ed3b75 at consistency null
2022-03-22 06:27:04.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1629970420-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:04.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [32315240-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@213d1449 at consistency null
2022-03-22 06:27:04.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [32315240-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:04.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553957789-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7595c506 at consistency null
2022-03-22 06:27:04.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553957789-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:04.335 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1339611373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4293848c at consistency null
2022-03-22 06:27:04.335 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1339611373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:04.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [349959709-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23442938 at consistency null
2022-03-22 06:27:04.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [349959709-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:04.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103764409-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a12fe45 at consistency null
2022-03-22 06:27:04.823 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103764409-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:05.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564934883-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31f4bbf4 at consistency null
2022-03-22 06:27:05.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564934883-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:05.232 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1423029782-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f9deea2 at consistency null
2022-03-22 06:27:05.233 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1423029782-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:05.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [535939471-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d121780 at consistency null
2022-03-22 06:27:05.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [535939471-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:06.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172483813-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@100f2cb1 at consistency null
2022-03-22 06:27:06.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1172483813-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:07.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542633120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b00e5f at consistency null
2022-03-22 06:27:07.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542633120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:07.314 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:27:12.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045747130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4581895 at consistency null
2022-03-22 06:27:12.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2045747130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:12.395 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757650105-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@790ba189 at consistency null
2022-03-22 06:27:12.395 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1757650105-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:12.455 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [939954882-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@108870bd at consistency null
2022-03-22 06:27:12.455 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [939954882-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:12.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [657152910-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14ce4605 at consistency null
2022-03-22 06:27:12.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [657152910-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:12.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [574362270-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@509678b9 at consistency null
2022-03-22 06:27:12.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [574362270-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:12.668 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1458467892-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@276b1768 at consistency null
2022-03-22 06:27:12.669 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1458467892-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:12.759 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1525836969-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@417393ee at consistency null
2022-03-22 06:27:12.759 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1525836969-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:12.886 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [670608210-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22c378c9 at consistency null
2022-03-22 06:27:12.886 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [670608210-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:13.090 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1695737297-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a487b9e at consistency null
2022-03-22 06:27:13.090 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1695737297-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:13.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081635475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@567e3e8 at consistency null
2022-03-22 06:27:13.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081635475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:13.337 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:27:13.352 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:27:13.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [161925393-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5856ec49 at consistency null
2022-03-22 06:27:13.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [161925393-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:14.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1810256204-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61eac2ba at consistency null
2022-03-22 06:27:14.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1810256204-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:14.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [422059404-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58c7d192 at consistency null
2022-03-22 06:27:14.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [422059404-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:15.337 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [440867024-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@628511c1 at consistency null
2022-03-22 06:27:15.337 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [440867024-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:16.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [630629576-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@dde9d7c at consistency null
2022-03-22 06:27:16.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [630629576-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:16.146 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:27:21.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2082729223-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a16616f at consistency null
2022-03-22 06:27:21.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2082729223-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:21.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633232242-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cbb7cfe at consistency null
2022-03-22 06:27:21.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633232242-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:21.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [127747725-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@167a65b1 at consistency null
2022-03-22 06:27:21.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [127747725-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:21.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389874258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f363f7f at consistency null
2022-03-22 06:27:21.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [389874258-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:21.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1947569767-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@662226ef at consistency null
2022-03-22 06:27:21.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1947569767-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:21.849 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598163036-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61f9ea59 at consistency null
2022-03-22 06:27:21.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598163036-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:22.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315159435-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24eb5c3a at consistency null
2022-03-22 06:27:22.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1315159435-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:22.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825428359-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a78521f at consistency null
2022-03-22 06:27:22.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [825428359-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:23.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1762516848-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d9a16e1 at consistency null
2022-03-22 06:27:23.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1762516848-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:23.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1933728211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@239f8e29 at consistency null
2022-03-22 06:27:23.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1933728211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:24.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [547241429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65bb45e4 at consistency null
2022-03-22 06:27:24.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [547241429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:24.251 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:27:29.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155744112-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38cb6aac at consistency null
2022-03-22 06:27:29.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [155744112-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:29.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [247058294-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67fa555a at consistency null
2022-03-22 06:27:29.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [247058294-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:29.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1867813811-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6273b925 at consistency null
2022-03-22 06:27:29.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1867813811-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:29.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1056271129-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dd844ea at consistency null
2022-03-22 06:27:29.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1056271129-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:29.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959553088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ea5dbf7 at consistency null
2022-03-22 06:27:29.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1959553088-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:29.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [877579216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b4019fc at consistency null
2022-03-22 06:27:29.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [877579216-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:29.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2144083937-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74053320 at consistency null
2022-03-22 06:27:29.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2144083937-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:30.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794379426-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b65c65e at consistency null
2022-03-22 06:27:30.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794379426-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:30.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2112742868-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b8fd2a6 at consistency null
2022-03-22 06:27:30.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2112742868-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:30.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1061134834-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@700bbbd2 at consistency null
2022-03-22 06:27:30.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1061134834-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:30.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [546538211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d4a0ac0 at consistency null
2022-03-22 06:27:30.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [546538211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:31.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1792543053-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68586e01 at consistency null
2022-03-22 06:27:31.001 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1792543053-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:31.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2022277767-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6dc79b4e at consistency null
2022-03-22 06:27:31.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2022277767-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:31.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981179477-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40bf780f at consistency null
2022-03-22 06:27:31.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [981179477-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:32.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [113476963-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@165d2f0e at consistency null
2022-03-22 06:27:32.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [113476963-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:32.721 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:27:37.755 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754677399-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4791cb60 at consistency null
2022-03-22 06:27:37.755 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754677399-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:37.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [416433205-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c55bc9d at consistency null
2022-03-22 06:27:37.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [416433205-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:37.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2125326926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3273bd33 at consistency null
2022-03-22 06:27:37.931 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2125326926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:38.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1683980389-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d29c12c at consistency null
2022-03-22 06:27:38.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1683980389-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:38.570 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [325524919-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42ac922 at consistency null
2022-03-22 06:27:38.570 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [325524919-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:39.380 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421400578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55ca3c6e at consistency null
2022-03-22 06:27:39.380 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421400578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:40.993 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [141588213-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7aff99c1 at consistency null
2022-03-22 06:27:40.993 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [141588213-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:40.994 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:27:43.366 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:27:43.370 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:27:46.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [271881014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79dc1713 at consistency null
2022-03-22 06:27:46.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [271881014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:46.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [991276578-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b6da2d3 at consistency null
2022-03-22 06:27:46.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [991276578-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:46.086 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1382168386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fc0d29c at consistency null
2022-03-22 06:27:46.086 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1382168386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:46.198 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504588837-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29777734 at consistency null
2022-03-22 06:27:46.198 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1504588837-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:46.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1832362245-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@308a8986 at consistency null
2022-03-22 06:27:46.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1832362245-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:46.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2076534335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7637ce7e at consistency null
2022-03-22 06:27:46.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2076534335-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:46.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [954111522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f4ab616 at consistency null
2022-03-22 06:27:46.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [954111522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:47.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276402688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d96b41c at consistency null
2022-03-22 06:27:47.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276402688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:47.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1075193475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2934d291 at consistency null
2022-03-22 06:27:47.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1075193475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:47.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [280118304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6138bdf7 at consistency null
2022-03-22 06:27:47.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [280118304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:48.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553051538-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b485dc0 at consistency null
2022-03-22 06:27:48.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1553051538-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:49.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [10805138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@130632e1 at consistency null
2022-03-22 06:27:49.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [10805138-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:49.811 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:27:54.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903678267-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ca9ca4a at consistency null
2022-03-22 06:27:54.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903678267-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:54.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936067185-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@385a1b2d at consistency null
2022-03-22 06:27:54.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1936067185-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:55.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [576374448-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11d5e7bb at consistency null
2022-03-22 06:27:55.019 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [576374448-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:55.136 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [77911334-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33664e5a at consistency null
2022-03-22 06:27:55.136 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [77911334-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:55.309 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267015810-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61cf29e6 at consistency null
2022-03-22 06:27:55.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1267015810-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:55.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [141794817-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9a4d581 at consistency null
2022-03-22 06:27:55.519 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [141794817-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:56.030 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095621881-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37f21331 at consistency null
2022-03-22 06:27:56.031 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095621881-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:56.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1581728167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36e79f0b at consistency null
2022-03-22 06:27:56.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1581728167-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:56.939 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1017441245-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e7f0028 at consistency null
2022-03-22 06:27:56.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1017441245-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:57.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674490846-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d8d0bb at consistency null
2022-03-22 06:27:57.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [674490846-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:58.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299191631-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2614302a at consistency null
2022-03-22 06:27:58.615 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [299191631-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:27:58.618 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:28:03.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1971023019-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c01ec24 at consistency null
2022-03-22 06:28:03.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1971023019-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:03.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1847237594-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7aa9d7cc at consistency null
2022-03-22 06:28:03.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1847237594-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:03.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1498473182-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3db39d2a at consistency null
2022-03-22 06:28:03.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1498473182-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:04.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1658090382-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3aa17fd6 at consistency null
2022-03-22 06:28:04.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1658090382-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:04.067 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281639755-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19e2d46a at consistency null
2022-03-22 06:28:04.068 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281639755-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:04.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989854038-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d41772d at consistency null
2022-03-22 06:28:04.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989854038-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:04.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [679647561-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c3af562 at consistency null
2022-03-22 06:28:04.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [679647561-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:04.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1770772436-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d5474fc at consistency null
2022-03-22 06:28:04.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1770772436-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:05.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [128682419-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@321aa6b8 at consistency null
2022-03-22 06:28:05.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [128682419-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:05.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [663452259-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@144457c9 at consistency null
2022-03-22 06:28:05.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [663452259-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:06.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [469759754-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f8e21c6 at consistency null
2022-03-22 06:28:06.020 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [469759754-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:06.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754820822-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51634f5f at consistency null
2022-03-22 06:28:06.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [754820822-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:06.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2000700651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@362b0cad at consistency null
2022-03-22 06:28:06.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2000700651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:06.937 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:28:11.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199516162-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e7efbe at consistency null
2022-03-22 06:28:11.948 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199516162-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:12.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1875345875-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e597e75 at consistency null
2022-03-22 06:28:12.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1875345875-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:12.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776819179-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@750bd756 at consistency null
2022-03-22 06:28:12.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [776819179-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:12.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1541353429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b2bf33e at consistency null
2022-03-22 06:28:12.147 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1541353429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:12.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1930666204-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@248e75fa at consistency null
2022-03-22 06:28:12.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1930666204-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:12.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [545877620-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51d10dc8 at consistency null
2022-03-22 06:28:12.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [545877620-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:12.581 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1678750087-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a4bc1c3 at consistency null
2022-03-22 06:28:12.581 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1678750087-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:12.995 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1183615828-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ebd6fb7 at consistency null
2022-03-22 06:28:12.996 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1183615828-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:13.397 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:28:13.412 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:28:13.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [687326745-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69cbfc3f at consistency null
2022-03-22 06:28:13.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [687326745-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:13.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1139206077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@302d1383 at consistency null
2022-03-22 06:28:13.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1139206077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:14.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1526523343-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14cc999c at consistency null
2022-03-22 06:28:14.250 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1526523343-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:14.659 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2022179823-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41598c65 at consistency null
2022-03-22 06:28:14.659 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2022179823-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:15.067 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851377129-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77c5aa75 at consistency null
2022-03-22 06:28:15.067 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1851377129-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:15.881 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [452336784-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fac0e05 at consistency null
2022-03-22 06:28:15.882 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [452336784-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:15.883 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:28:20.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [655970460-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fdc7215 at consistency null
2022-03-22 06:28:20.928 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [655970460-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:21.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532068587-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@526be72f at consistency null
2022-03-22 06:28:21.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [532068587-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:21.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063272569-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8276915 at consistency null
2022-03-22 06:28:21.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1063272569-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:21.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366533017-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47b49232 at consistency null
2022-03-22 06:28:21.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [366533017-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:21.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2024338958-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b440c0b at consistency null
2022-03-22 06:28:21.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2024338958-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:21.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383299088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45c77ed4 at consistency null
2022-03-22 06:28:21.412 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383299088-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:21.609 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1873648047-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2601515c at consistency null
2022-03-22 06:28:21.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1873648047-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:21.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013613010-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dfe35c3 at consistency null
2022-03-22 06:28:21.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1013613010-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:22.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [63772887-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@553b11e9 at consistency null
2022-03-22 06:28:22.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [63772887-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:22.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1253524294-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b74e444 at consistency null
2022-03-22 06:28:22.330 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1253524294-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:22.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1230293854-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@753cbc7 at consistency null
2022-03-22 06:28:22.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1230293854-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:23.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [559458231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b8fb6c3 at consistency null
2022-03-22 06:28:23.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [559458231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:23.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1253578987-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6364a02 at consistency null
2022-03-22 06:28:23.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1253578987-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:24.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [988139401-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a6cc844 at consistency null
2022-03-22 06:28:24.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [988139401-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:24.326 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:28:29.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1434774960-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13f8273d at consistency null
2022-03-22 06:28:29.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1434774960-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:29.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [830836136-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6537b22f at consistency null
2022-03-22 06:28:29.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [830836136-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:29.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [997237811-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@516592d1 at consistency null
2022-03-22 06:28:29.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [997237811-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:29.670 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [796769619-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3da74e51 at consistency null
2022-03-22 06:28:29.670 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [796769619-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:29.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1566977121-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fb44717 at consistency null
2022-03-22 06:28:29.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1566977121-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:29.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074987688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@374a3604 at consistency null
2022-03-22 06:28:29.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1074987688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:29.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2130474917-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a44c926 at consistency null
2022-03-22 06:28:29.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2130474917-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:29.966 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1009371660-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d96d3f9 at consistency null
2022-03-22 06:28:29.966 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1009371660-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:30.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1346432204-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57a9a3 at consistency null
2022-03-22 06:28:30.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1346432204-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:30.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136225326-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@664b45bb at consistency null
2022-03-22 06:28:30.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [136225326-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:30.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365084119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a2aadad at consistency null
2022-03-22 06:28:30.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365084119-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:30.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741257929-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71d8ae0a at consistency null
2022-03-22 06:28:30.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741257929-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:30.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [817416816-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c7f0b23 at consistency null
2022-03-22 06:28:30.906 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [817416816-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:31.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1296695707-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70a5e98a at consistency null
2022-03-22 06:28:31.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1296695707-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:32.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1306927584-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f3d65f6 at consistency null
2022-03-22 06:28:32.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1306927584-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:32.214 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:28:37.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1634018209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b2126e5 at consistency null
2022-03-22 06:28:37.240 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1634018209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:37.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402165431-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f3310b1 at consistency null
2022-03-22 06:28:37.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402165431-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:37.334 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [178459402-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70854564 at consistency null
2022-03-22 06:28:37.334 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [178459402-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:37.397 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1991695007-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34d38c57 at consistency null
2022-03-22 06:28:37.397 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1991695007-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:37.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1049008067-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26d79226 at consistency null
2022-03-22 06:28:37.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1049008067-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:37.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1842887570-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11d6b8d5 at consistency null
2022-03-22 06:28:37.555 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1842887570-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:37.661 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146520672-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b2261ee at consistency null
2022-03-22 06:28:37.661 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146520672-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:37.880 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070792995-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@633bfde9 at consistency null
2022-03-22 06:28:37.881 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1070792995-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:38.288 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [895594901-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79e7150a at consistency null
2022-03-22 06:28:38.288 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [895594901-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:38.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [432605184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dd30af3 at consistency null
2022-03-22 06:28:38.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [432605184-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:39.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1824442135-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@394e4c4e at consistency null
2022-03-22 06:28:39.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1824442135-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:39.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1271077306-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78c05128 at consistency null
2022-03-22 06:28:39.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1271077306-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:39.925 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:28:43.424 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:28:43.442 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:28:44.943 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1939268686-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d014851 at consistency null
2022-03-22 06:28:44.944 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1939268686-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:45.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575568692-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@97e99da at consistency null
2022-03-22 06:28:45.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575568692-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:45.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [953325077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42998887 at consistency null
2022-03-22 06:28:45.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [953325077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:45.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1804551804-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38aac8ed at consistency null
2022-03-22 06:28:45.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1804551804-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:45.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [796883927-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fde5c69 at consistency null
2022-03-22 06:28:45.322 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [796883927-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:45.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117673687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7293b401 at consistency null
2022-03-22 06:28:45.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117673687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:45.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383559483-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c40d764 at consistency null
2022-03-22 06:28:45.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383559483-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:45.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129996719-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f03ae94 at consistency null
2022-03-22 06:28:45.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129996719-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:46.278 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397351888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@df3cc9 at consistency null
2022-03-22 06:28:46.278 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [397351888-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:47.085 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540523624-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a5e3041 at consistency null
2022-03-22 06:28:47.085 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [540523624-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:48.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1226895356-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@743e4c36 at consistency null
2022-03-22 06:28:48.725 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1226895356-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:48.727 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:28:53.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499969646-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bb2b2b at consistency null
2022-03-22 06:28:53.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1499969646-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:53.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [922306825-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f7b6641 at consistency null
2022-03-22 06:28:53.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [922306825-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:54.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732981058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72753098 at consistency null
2022-03-22 06:28:54.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732981058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:54.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1696670407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d48af8e at consistency null
2022-03-22 06:28:54.505 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1696670407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:54.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23151301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9259d08 at consistency null
2022-03-22 06:28:54.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [23151301-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:55.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100093665-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19f76bde at consistency null
2022-03-22 06:28:55.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2100093665-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:55.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1435373681-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42dd7988 at consistency null
2022-03-22 06:28:55.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1435373681-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:55.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [970274589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a019a82 at consistency null
2022-03-22 06:28:55.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [970274589-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:56.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [841559432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f699f17 at consistency null
2022-03-22 06:28:56.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [841559432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:57.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671673030-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2de2acbf at consistency null
2022-03-22 06:28:57.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [671673030-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:28:57.314 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:29:02.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220228142-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45a81b2 at consistency null
2022-03-22 06:29:02.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [220228142-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:02.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [501074248-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2229ce31 at consistency null
2022-03-22 06:29:02.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [501074248-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:02.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1717821974-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61fb7c44 at consistency null
2022-03-22 06:29:02.502 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1717821974-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:02.604 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [164788934-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5944a390 at consistency null
2022-03-22 06:29:02.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [164788934-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:02.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [120393928-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@943ee1b at consistency null
2022-03-22 06:29:02.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [120393928-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:02.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950949100-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6577f144 at consistency null
2022-03-22 06:29:02.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950949100-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:03.439 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [230895905-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fd4b8f2 at consistency null
2022-03-22 06:29:03.440 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [230895905-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:04.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391998997-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23f0d8bb at consistency null
2022-03-22 06:29:04.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [391998997-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:06.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2014943078-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35d5c9e8 at consistency null
2022-03-22 06:29:06.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2014943078-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:06.138 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:29:11.171 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151027447-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ff2f3c7 at consistency null
2022-03-22 06:29:11.171 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1151027447-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:11.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1346708348-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@159bc5e3 at consistency null
2022-03-22 06:29:11.251 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1346708348-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:11.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [747946663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@308bf2eb at consistency null
2022-03-22 06:29:11.403 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [747946663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:11.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402838260-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b0ae79b at consistency null
2022-03-22 06:29:11.517 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [402838260-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:11.631 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [113307123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@539e45f6 at consistency null
2022-03-22 06:29:11.631 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [113307123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:11.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2145869972-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18e136bb at consistency null
2022-03-22 06:29:11.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2145869972-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:12.254 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1970013237-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@779283e6 at consistency null
2022-03-22 06:29:12.254 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1970013237-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:13.073 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2071461150-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@bd884b0 at consistency null
2022-03-22 06:29:13.073 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2071461150-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:13.454 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:29:13.457 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:29:14.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052663542-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a48c1c8 at consistency null
2022-03-22 06:29:14.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052663542-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:14.729 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:29:19.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1048450281-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b0d7ed4 at consistency null
2022-03-22 06:29:19.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1048450281-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:19.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732361407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@586347e0 at consistency null
2022-03-22 06:29:19.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [732361407-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:20.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1878881752-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1751a558 at consistency null
2022-03-22 06:29:20.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1878881752-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:20.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1557200457-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b7b60fb at consistency null
2022-03-22 06:29:20.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1557200457-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:20.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315060326-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bd04a9e at consistency null
2022-03-22 06:29:20.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [315060326-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:20.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [241720203-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@694e5443 at consistency null
2022-03-22 06:29:20.411 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [241720203-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:20.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180337707-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73af5757 at consistency null
2022-03-22 06:29:20.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180337707-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:21.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1363556739-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53739d61 at consistency null
2022-03-22 06:29:21.109 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1363556739-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:21.315 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2024486797-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6835f0f2 at consistency null
2022-03-22 06:29:21.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2024486797-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:21.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [877895440-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27e2e000 at consistency null
2022-03-22 06:29:21.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [877895440-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:22.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1132082005-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28fa8236 at consistency null
2022-03-22 06:29:22.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1132082005-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:22.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596672664-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c97328 at consistency null
2022-03-22 06:29:22.421 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [596672664-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:23.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [36101176-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@766eaba4 at consistency null
2022-03-22 06:29:23.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [36101176-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:23.320 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:29:28.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [715483251-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a797c6d at consistency null
2022-03-22 06:29:28.419 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [715483251-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:28.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [945231593-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40e30e39 at consistency null
2022-03-22 06:29:28.531 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [945231593-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:28.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [535086395-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63701f0f at consistency null
2022-03-22 06:29:28.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [535086395-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:28.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [722935267-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6040ab4e at consistency null
2022-03-22 06:29:28.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [722935267-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:28.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720375298-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@219b293f at consistency null
2022-03-22 06:29:28.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [720375298-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:29.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [714126827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@795179f3 at consistency null
2022-03-22 06:29:29.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [714126827-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:29.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [325349535-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a3a99b1 at consistency null
2022-03-22 06:29:29.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [325349535-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:29.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1023765-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f454b63 at consistency null
2022-03-22 06:29:29.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1023765-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:30.053 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [707068168-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76f5765 at consistency null
2022-03-22 06:29:30.053 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [707068168-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:30.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [87410250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7503289a at consistency null
2022-03-22 06:29:30.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [87410250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:30.627 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [895860475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bbad09a at consistency null
2022-03-22 06:29:30.627 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [895860475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:30.853 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1240365173-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e718add at consistency null
2022-03-22 06:29:30.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1240365173-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:31.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1262558465-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13d82c09 at consistency null
2022-03-22 06:29:31.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1262558465-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:31.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1617794485-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1220b064 at consistency null
2022-03-22 06:29:31.730 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1617794485-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:31.731 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:29:36.739 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2028436773-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70e5e79f at consistency null
2022-03-22 06:29:36.740 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2028436773-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:36.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1123065756-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@228f35ee at consistency null
2022-03-22 06:29:36.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1123065756-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:36.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [26885078-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4117cbbe at consistency null
2022-03-22 06:29:36.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [26885078-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:36.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [232179309-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5623e068 at consistency null
2022-03-22 06:29:36.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [232179309-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:37.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1310428327-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9a52c2c at consistency null
2022-03-22 06:29:37.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1310428327-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:37.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139589255-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@525e48b3 at consistency null
2022-03-22 06:29:37.137 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [139589255-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:37.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1489320144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d6b237d at consistency null
2022-03-22 06:29:37.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1489320144-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:37.337 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662740491-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6402b618 at consistency null
2022-03-22 06:29:37.337 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1662740491-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:37.569 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1912061563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44e622f8 at consistency null
2022-03-22 06:29:37.569 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1912061563-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:38.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989516044-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64ddf107 at consistency null
2022-03-22 06:29:38.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989516044-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:38.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [124206774-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1059562a at consistency null
2022-03-22 06:29:38.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [124206774-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:39.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1057248609-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@611a904f at consistency null
2022-03-22 06:29:39.449 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1057248609-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:39.450 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:29:43.486 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:29:43.500 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:29:44.461 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1942761628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@141e9f76 at consistency null
2022-03-22 06:29:44.461 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1942761628-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:44.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [702257170-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f40013b at consistency null
2022-03-22 06:29:44.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [702257170-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:44.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [393711889-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4572ecff at consistency null
2022-03-22 06:29:44.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [393711889-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:44.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180711875-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70df84f6 at consistency null
2022-03-22 06:29:44.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [180711875-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:45.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [404748503-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10da50b4 at consistency null
2022-03-22 06:29:45.062 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [404748503-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:45.280 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836181710-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f8cc24d at consistency null
2022-03-22 06:29:45.281 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1836181710-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:45.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1185219432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dcbc257 at consistency null
2022-03-22 06:29:45.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1185219432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:45.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1733241110-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@513f052c at consistency null
2022-03-22 06:29:45.916 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1733241110-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:46.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [7449704-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13a2a04f at consistency null
2022-03-22 06:29:46.334 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [7449704-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:47.273 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1654915888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a5586dc at consistency null
2022-03-22 06:29:47.273 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1654915888-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:48.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208986703-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@798f3abb at consistency null
2022-03-22 06:29:48.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208986703-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:48.113 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:29:53.219 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [317549902-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@503056c3 at consistency null
2022-03-22 06:29:53.220 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [317549902-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:53.317 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [527602071-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56162452 at consistency null
2022-03-22 06:29:53.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [527602071-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:53.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1561181386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@233b2a at consistency null
2022-03-22 06:29:53.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1561181386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:53.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1983925182-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f99377 at consistency null
2022-03-22 06:29:53.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1983925182-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:53.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264167205-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52ad5be5 at consistency null
2022-03-22 06:29:53.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [264167205-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:54.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72026476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35fec156 at consistency null
2022-03-22 06:29:54.006 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72026476-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:54.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [881856195-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c9d46a8 at consistency null
2022-03-22 06:29:54.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [881856195-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:54.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090104916-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60330a67 at consistency null
2022-03-22 06:29:54.737 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1090104916-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:55.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [960960153-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58f917f7 at consistency null
2022-03-22 06:29:55.210 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [960960153-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:56.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1696295092-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b4e99cf at consistency null
2022-03-22 06:29:56.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1696295092-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:56.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1363772610-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d203408 at consistency null
2022-03-22 06:29:56.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1363772610-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:29:56.907 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:30:01.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1548684209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25387df9 at consistency null
2022-03-22 06:30:01.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1548684209-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:02.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564294527-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35ee8006 at consistency null
2022-03-22 06:30:02.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1564294527-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:02.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667080356-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a01e795 at consistency null
2022-03-22 06:30:02.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1667080356-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:02.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117031833-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65d18a9 at consistency null
2022-03-22 06:30:02.228 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2117031833-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:02.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1520028014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60ec6a2e at consistency null
2022-03-22 06:30:02.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1520028014-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:02.726 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1861992839-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21432885 at consistency null
2022-03-22 06:30:02.727 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1861992839-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:03.149 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1613567100-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@742b4e77 at consistency null
2022-03-22 06:30:03.150 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1613567100-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:04.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1689912447-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1add980b at consistency null
2022-03-22 06:30:04.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1689912447-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:04.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887204159-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19e32c64 at consistency null
2022-03-22 06:30:04.834 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1887204159-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:05.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1077633870-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@215c28dd at consistency null
2022-03-22 06:30:05.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1077633870-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:05.716 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:30:10.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211961793-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@faa9277 at consistency null
2022-03-22 06:30:10.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1211961793-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:10.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2089839941-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79c14e49 at consistency null
2022-03-22 06:30:10.751 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2089839941-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:10.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1868962915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7941cd19 at consistency null
2022-03-22 06:30:10.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1868962915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:10.885 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1960588242-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@391b182c at consistency null
2022-03-22 06:30:10.886 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1960588242-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:11.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1414502289-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62790c52 at consistency null
2022-03-22 06:30:11.046 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1414502289-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:11.264 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [734667288-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11b9b552 at consistency null
2022-03-22 06:30:11.264 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [734667288-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:11.473 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1955636742-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66e79b79 at consistency null
2022-03-22 06:30:11.473 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1955636742-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:11.885 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1632066021-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dc35f1 at consistency null
2022-03-22 06:30:11.885 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1632066021-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:12.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235007811-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d6da2c4 at consistency null
2022-03-22 06:30:12.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235007811-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:13.513 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:30:13.521 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:30:14.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859605130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71a2911e at consistency null
2022-03-22 06:30:14.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [859605130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:14.307 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:30:19.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1122875517-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bc4e000 at consistency null
2022-03-22 06:30:19.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1122875517-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:19.513 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [423512500-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dd13b34 at consistency null
2022-03-22 06:30:19.514 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [423512500-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:19.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [902086001-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@deadbc8 at consistency null
2022-03-22 06:30:19.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [902086001-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:19.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1651182491-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6825810a at consistency null
2022-03-22 06:30:19.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1651182491-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:19.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298823675-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@646da8ee at consistency null
2022-03-22 06:30:19.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1298823675-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:20.052 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [653453835-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@202127cc at consistency null
2022-03-22 06:30:20.052 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [653453835-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:20.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [911074449-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bce9c9b at consistency null
2022-03-22 06:30:20.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [911074449-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:20.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [205522448-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fa76f39 at consistency null
2022-03-22 06:30:20.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [205522448-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:20.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [318440386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36049afb at consistency null
2022-03-22 06:30:20.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [318440386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:21.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [594022692-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fb2d4b0 at consistency null
2022-03-22 06:30:21.026 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [594022692-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:21.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052591796-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@144dc215 at consistency null
2022-03-22 06:30:21.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2052591796-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:21.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982130537-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60c5125 at consistency null
2022-03-22 06:30:21.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1982130537-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:22.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1007358332-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f920200 at consistency null
2022-03-22 06:30:22.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1007358332-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:22.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1602868968-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2264291a at consistency null
2022-03-22 06:30:22.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1602868968-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:22.904 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:30:28.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1710137373-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58430a29 at consistency null
2022-03-22 06:30:28.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1710137373-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:28.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [886071912-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c22ed7d at consistency null
2022-03-22 06:30:28.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [886071912-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:28.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1353979263-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@233f6afb at consistency null
2022-03-22 06:30:28.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1353979263-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:28.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1092331287-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2090ab7f at consistency null
2022-03-22 06:30:28.348 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1092331287-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:28.476 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373964641-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6042ff06 at consistency null
2022-03-22 06:30:28.477 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1373964641-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:28.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1720836631-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b20af76 at consistency null
2022-03-22 06:30:28.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1720836631-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:28.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [385748055-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13106c5b at consistency null
2022-03-22 06:30:28.923 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [385748055-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:29.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [172040832-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@367ee615 at consistency null
2022-03-22 06:30:29.408 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [172040832-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:30.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1939246248-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41339513 at consistency null
2022-03-22 06:30:30.212 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1939246248-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:31.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [444488087-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fc40e47 at consistency null
2022-03-22 06:30:31.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [444488087-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:31.948 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:30:37.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2089054779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e4b0de4 at consistency null
2022-03-22 06:30:37.038 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2089054779-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:37.081 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2144471561-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38d8033 at consistency null
2022-03-22 06:30:37.081 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2144471561-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:37.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [954612375-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a585537 at consistency null
2022-03-22 06:30:37.121 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [954612375-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:37.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1354530557-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d1c20d at consistency null
2022-03-22 06:30:37.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1354530557-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:37.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [267832187-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@373db7e at consistency null
2022-03-22 06:30:37.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [267832187-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:37.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620773582-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c104f3b at consistency null
2022-03-22 06:30:37.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1620773582-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:37.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1763747528-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14994e63 at consistency null
2022-03-22 06:30:37.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1763747528-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:37.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524135559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f533757 at consistency null
2022-03-22 06:30:37.921 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1524135559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:38.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674367924-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@266a7e8 at consistency null
2022-03-22 06:30:38.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1674367924-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:38.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1530488953-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c615908 at consistency null
2022-03-22 06:30:38.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1530488953-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:39.668 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [927175472-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32d5e8b1 at consistency null
2022-03-22 06:30:39.668 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [927175472-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:39.669 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:30:43.541 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:30:43.554 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:30:44.688 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [775337745-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@343a8833 at consistency null
2022-03-22 06:30:44.688 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [775337745-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:44.761 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [727879696-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35430bb at consistency null
2022-03-22 06:30:44.761 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [727879696-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:44.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998789029-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@659b80c0 at consistency null
2022-03-22 06:30:44.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [998789029-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:44.897 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276402679-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78967ca2 at consistency null
2022-03-22 06:30:44.897 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276402679-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:45.071 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [191640278-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@180d0893 at consistency null
2022-03-22 06:30:45.071 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [191640278-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:45.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [884064699-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a7e32c1 at consistency null
2022-03-22 06:30:45.295 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [884064699-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:45.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1304253935-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35a38ce at consistency null
2022-03-22 06:30:45.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1304253935-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:45.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [286757766-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4862276b at consistency null
2022-03-22 06:30:45.720 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [286757766-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:46.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2072508912-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@385bf764 at consistency null
2022-03-22 06:30:46.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2072508912-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:46.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098990370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73f82294 at consistency null
2022-03-22 06:30:46.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1098990370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:47.013 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1723102416-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21b6f86d at consistency null
2022-03-22 06:30:47.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1723102416-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:47.830 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1259491363-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78b6ca4f at consistency null
2022-03-22 06:30:47.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1259491363-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:47.832 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:30:52.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [262039752-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fb1e2ea at consistency null
2022-03-22 06:30:52.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [262039752-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:53.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1145942602-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2def8f0a at consistency null
2022-03-22 06:30:53.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1145942602-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:53.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [26889447-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b8c7f4a at consistency null
2022-03-22 06:30:53.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [26889447-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:53.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1490065343-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75263ec6 at consistency null
2022-03-22 06:30:53.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1490065343-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:53.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [611613765-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3456704d at consistency null
2022-03-22 06:30:53.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [611613765-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:53.977 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [218256739-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cf83fa2 at consistency null
2022-03-22 06:30:53.978 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [218256739-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:54.221 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579998544-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28ececf5 at consistency null
2022-03-22 06:30:54.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1579998544-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:54.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [35505221-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cb9ac36 at consistency null
2022-03-22 06:30:54.512 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [35505221-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:54.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [93066775-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b6a69f4 at consistency null
2022-03-22 06:30:54.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [93066775-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:55.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [440066696-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66c24a14 at consistency null
2022-03-22 06:30:55.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [440066696-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:56.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [574909122-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2986997d at consistency null
2022-03-22 06:30:56.311 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [574909122-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:30:56.319 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:31:01.325 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1310178595-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28df5867 at consistency null
2022-03-22 06:31:01.326 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1310178595-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:01.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [734930520-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30e2a141 at consistency null
2022-03-22 06:31:01.407 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [734930520-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:01.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1722662667-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4484cfa3 at consistency null
2022-03-22 06:31:01.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1722662667-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:01.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1740557158-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59d4fb3f at consistency null
2022-03-22 06:31:01.602 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1740557158-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:01.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [347311127-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b9a8c02 at consistency null
2022-03-22 06:31:01.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [347311127-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:01.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318425663-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ed7cc99 at consistency null
2022-03-22 06:31:01.930 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318425663-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:02.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1162559233-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c4cf737 at consistency null
2022-03-22 06:31:02.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1162559233-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:02.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1421814069-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32bfe082 at consistency null
2022-03-22 06:31:02.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1421814069-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:02.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1558496385-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d9a804b at consistency null
2022-03-22 06:31:02.415 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1558496385-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:02.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051887438-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d392a65 at consistency null
2022-03-22 06:31:02.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1051887438-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:03.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [572331182-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56678fba at consistency null
2022-03-22 06:31:03.108 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [572331182-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:03.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256276313-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b11ce04 at consistency null
2022-03-22 06:31:03.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256276313-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:03.913 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:31:08.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1971092220-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22b09df5 at consistency null
2022-03-22 06:31:08.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1971092220-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:08.946 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1608391658-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16eee1c4 at consistency null
2022-03-22 06:31:08.946 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1608391658-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:08.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [355261659-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c9eb6fc at consistency null
2022-03-22 06:31:08.983 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [355261659-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:09.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703368600-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69ea2685 at consistency null
2022-03-22 06:31:09.036 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [703368600-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:09.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1062316483-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f935178 at consistency null
2022-03-22 06:31:09.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1062316483-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:09.248 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276821128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e92698f at consistency null
2022-03-22 06:31:09.248 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [276821128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:09.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1665542232-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27b213f7 at consistency null
2022-03-22 06:31:09.355 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1665542232-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:09.562 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [772532709-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fd4c78a at consistency null
2022-03-22 06:31:09.562 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [772532709-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:09.985 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472273494-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a995d99 at consistency null
2022-03-22 06:31:09.985 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [472273494-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:10.398 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1938960110-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77fa8b54 at consistency null
2022-03-22 06:31:10.398 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1938960110-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:10.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [566549548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@406003f7 at consistency null
2022-03-22 06:31:10.810 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [566549548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:11.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349602608-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79b164e6 at consistency null
2022-03-22 06:31:11.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1349602608-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:12.044 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2120946058-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79c49bdc at consistency null
2022-03-22 06:31:12.045 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2120946058-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:12.046 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:31:13.572 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:31:13.578 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:31:17.050 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [485275127-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ac70f9b at consistency null
2022-03-22 06:31:17.050 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [485275127-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:17.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458113018-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a78c08f at consistency null
2022-03-22 06:31:17.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [458113018-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:17.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099753838-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@689c1172 at consistency null
2022-03-22 06:31:17.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1099753838-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:17.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741510548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@355b6dc6 at consistency null
2022-03-22 06:31:17.304 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1741510548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:17.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [511120791-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57820721 at consistency null
2022-03-22 06:31:17.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [511120791-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:17.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957451852-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ba2c9e6 at consistency null
2022-03-22 06:31:17.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957451852-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:17.858 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1389172782-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bd5e61 at consistency null
2022-03-22 06:31:17.859 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1389172782-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:18.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1605473300-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d66368d at consistency null
2022-03-22 06:31:18.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1605473300-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:18.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2035083736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a9a9f0f at consistency null
2022-03-22 06:31:18.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2035083736-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:19.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1874112645-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@376a9e43 at consistency null
2022-03-22 06:31:19.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1874112645-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:19.526 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:31:24.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1434634549-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ecd9099 at consistency null
2022-03-22 06:31:24.619 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1434634549-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:24.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [104920741-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c31793 at consistency null
2022-03-22 06:31:24.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [104920741-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:24.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [408739228-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cdf8223 at consistency null
2022-03-22 06:31:24.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [408739228-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:24.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1378252426-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fa4837d at consistency null
2022-03-22 06:31:24.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1378252426-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:25.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [288282266-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ddf9148 at consistency null
2022-03-22 06:31:25.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [288282266-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:25.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [282783684-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@373a64ef at consistency null
2022-03-22 06:31:25.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [282783684-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:25.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1855221886-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37f426b1 at consistency null
2022-03-22 06:31:25.201 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1855221886-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:25.413 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258459904-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c70e459 at consistency null
2022-03-22 06:31:25.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258459904-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:25.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1746556016-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5df4714e at consistency null
2022-03-22 06:31:25.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1746556016-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:25.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804856440-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@398d7b03 at consistency null
2022-03-22 06:31:25.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [804856440-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:26.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1646502009-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f5a132d at consistency null
2022-03-22 06:31:26.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1646502009-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:26.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396525812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@240368b3 at consistency null
2022-03-22 06:31:26.710 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1396525812-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:27.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1774071095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a83137c at consistency null
2022-03-22 06:31:27.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1774071095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:28.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254945434-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8843e4c at consistency null
2022-03-22 06:31:28.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254945434-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:28.428 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:31:33.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886216557-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77b8970a at consistency null
2022-03-22 06:31:33.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1886216557-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:33.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1237249918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48c12fbe at consistency null
2022-03-22 06:31:33.515 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1237249918-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:33.627 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1374794987-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a1bcd29 at consistency null
2022-03-22 06:31:33.627 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1374794987-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:33.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1327050935-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@107737cd at consistency null
2022-03-22 06:31:33.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1327050935-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:33.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415857712-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@518a46a2 at consistency null
2022-03-22 06:31:33.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415857712-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:34.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1788839645-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5acab3f at consistency null
2022-03-22 06:31:34.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1788839645-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:34.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907354505-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cfe8c6 at consistency null
2022-03-22 06:31:34.405 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [907354505-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:34.612 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [814691188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@282f6e6d at consistency null
2022-03-22 06:31:34.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [814691188-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:35.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1177331347-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b13929c at consistency null
2022-03-22 06:31:35.029 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1177331347-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:35.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1351842277-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b063192 at consistency null
2022-03-22 06:31:35.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1351842277-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:36.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1326781207-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@357179ec at consistency null
2022-03-22 06:31:36.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1326781207-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:36.602 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:31:41.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [105422098-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66fe2529 at consistency null
2022-03-22 06:31:41.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [105422098-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:41.656 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636730230-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51cecc18 at consistency null
2022-03-22 06:31:41.657 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [636730230-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:41.689 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860477564-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bbeed12 at consistency null
2022-03-22 06:31:41.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1860477564-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:41.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [160340252-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15be5f90 at consistency null
2022-03-22 06:31:41.741 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [160340252-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:41.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1265382706-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27ee763 at consistency null
2022-03-22 06:31:41.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1265382706-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:41.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1937514274-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25f74538 at consistency null
2022-03-22 06:31:41.907 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1937514274-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:41.997 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [943760499-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@797deae9 at consistency null
2022-03-22 06:31:41.997 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [943760499-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:42.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1165724702-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ed87889 at consistency null
2022-03-22 06:31:42.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1165724702-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:42.166 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1872032806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b68d1ce at consistency null
2022-03-22 06:31:42.167 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1872032806-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:42.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088920057-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7429871e at consistency null
2022-03-22 06:31:42.227 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088920057-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:42.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096561446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@629511d0 at consistency null
2022-03-22 06:31:42.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1096561446-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:42.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1482574959-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fd4313b at consistency null
2022-03-22 06:31:42.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1482574959-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:42.681 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270771846-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@121f71ae at consistency null
2022-03-22 06:31:42.681 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [270771846-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:43.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2144436944-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ab61584 at consistency null
2022-03-22 06:31:43.107 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2144436944-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:43.601 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:31:43.612 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:31:43.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1597266703-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60678a93 at consistency null
2022-03-22 06:31:43.926 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1597266703-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:44.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1965246898-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ea4324d at consistency null
2022-03-22 06:31:44.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1965246898-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:44.748 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:31:49.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025444870-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17dfbe36 at consistency null
2022-03-22 06:31:49.836 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025444870-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:49.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [185408707-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ddf937c at consistency null
2022-03-22 06:31:49.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [185408707-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:50.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989989006-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51b0b346 at consistency null
2022-03-22 06:31:50.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1989989006-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:50.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421358390-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e59d567 at consistency null
2022-03-22 06:31:50.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [421358390-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:50.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [799184885-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@649de494 at consistency null
2022-03-22 06:31:50.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [799184885-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:50.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1277853126-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1528df08 at consistency null
2022-03-22 06:31:50.313 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1277853126-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:50.450 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555226223-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41d66468 at consistency null
2022-03-22 06:31:50.450 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1555226223-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:50.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [780425748-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16473d8 at consistency null
2022-03-22 06:31:50.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [780425748-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:50.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [362966286-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a568e24 at consistency null
2022-03-22 06:31:50.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [362966286-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:51.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [175765041-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5955d0f9 at consistency null
2022-03-22 06:31:51.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [175765041-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:51.640 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081213111-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b520a9f at consistency null
2022-03-22 06:31:51.640 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1081213111-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:52.531 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286256215-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@98a569e at consistency null
2022-03-22 06:31:52.531 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1286256215-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:52.542 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:31:57.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256614097-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3793ce25 at consistency null
2022-03-22 06:31:57.620 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1256614097-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:57.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258721730-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40b61236 at consistency null
2022-03-22 06:31:57.738 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258721730-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:57.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1598822688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b8bee4d at consistency null
2022-03-22 06:31:57.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1598822688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:58.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1179456656-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@206e7904 at consistency null
2022-03-22 06:31:58.042 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1179456656-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:58.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1058294888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7972fd6a at consistency null
2022-03-22 06:31:58.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1058294888-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:58.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1854982559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5daf8d8f at consistency null
2022-03-22 06:31:58.605 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1854982559-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:58.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1370114164-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@eda7967 at consistency null
2022-03-22 06:31:58.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1370114164-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:59.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146254661-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42e48b1a at consistency null
2022-03-22 06:31:59.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2146254661-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:31:59.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644649436-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6306a53c at consistency null
2022-03-22 06:31:59.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1644649436-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:00.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179550770-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e4ae97a at consistency null
2022-03-22 06:32:00.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [179550770-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:00.613 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:32:05.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156875512-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fba7d3e at consistency null
2022-03-22 06:32:05.706 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156875512-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:05.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [534297473-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69e09b88 at consistency null
2022-03-22 06:32:05.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [534297473-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:06.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197066781-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f19eccd at consistency null
2022-03-22 06:32:06.004 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1197066781-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:06.255 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1353489134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56f1d36d at consistency null
2022-03-22 06:32:06.255 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1353489134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:06.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [350643321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a7e1700 at consistency null
2022-03-22 06:32:06.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [350643321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:06.739 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1320731095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67aff11c at consistency null
2022-03-22 06:32:06.740 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1320731095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:07.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1203303231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@172016a0 at consistency null
2022-03-22 06:32:07.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1203303231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:07.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1945598668-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f12c592 at consistency null
2022-03-22 06:32:07.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1945598668-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:07.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1419054439-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@352e1d67 at consistency null
2022-03-22 06:32:07.503 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1419054439-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:08.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1546373211-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d913dcf at consistency null
2022-03-22 06:32:08.024 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1546373211-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:08.441 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1544219114-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a13bbcb at consistency null
2022-03-22 06:32:08.441 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1544219114-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:09.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [342088125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63f4d74e at consistency null
2022-03-22 06:32:09.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [342088125-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:09.338 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:32:13.613 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:32:13.626 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:32:14.374 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1751191651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22f1526 at consistency null
2022-03-22 06:32:14.374 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1751191651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:14.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1779069988-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25452b4b at consistency null
2022-03-22 06:32:14.431 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1779069988-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:14.475 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1631884598-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@604ec60a at consistency null
2022-03-22 06:32:14.475 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1631884598-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:14.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809847954-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6766deca at consistency null
2022-03-22 06:32:14.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1809847954-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:14.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199418245-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d1a6125 at consistency null
2022-03-22 06:32:14.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [199418245-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:14.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [854597513-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11090ebe at consistency null
2022-03-22 06:32:14.860 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [854597513-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:15.265 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2010694084-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@761e403 at consistency null
2022-03-22 06:32:15.265 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2010694084-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:16.068 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [476519723-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5620faab at consistency null
2022-03-22 06:32:16.069 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [476519723-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:17.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [9775595-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18fdba03 at consistency null
2022-03-22 06:32:17.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [9775595-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:17.705 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:32:22.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [850092583-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41d83b5c at consistency null
2022-03-22 06:32:22.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [850092583-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:22.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [63718231-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49918e98 at consistency null
2022-03-22 06:32:22.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [63718231-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:22.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [669419584-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18b5d0a3 at consistency null
2022-03-22 06:32:22.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [669419584-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:23.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [236271398-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d6d4c14 at consistency null
2022-03-22 06:32:23.116 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [236271398-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:23.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1732691609-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6def983e at consistency null
2022-03-22 06:32:23.316 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1732691609-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:23.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [116516829-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28cf2e55 at consistency null
2022-03-22 06:32:23.420 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [116516829-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:23.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365349835-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ec9d327 at consistency null
2022-03-22 06:32:23.632 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [365349835-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:23.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [843936338-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19c33106 at consistency null
2022-03-22 06:32:23.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [843936338-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:24.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1668159628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@87c7468 at consistency null
2022-03-22 06:32:24.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1668159628-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:24.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1781056563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59905eb4 at consistency null
2022-03-22 06:32:24.422 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1781056563-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:24.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1226031662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@990fe4a at consistency null
2022-03-22 06:32:24.701 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1226031662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:25.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269870780-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9dba57f at consistency null
2022-03-22 06:32:25.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [269870780-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:25.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [53229130-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@626f5f18 at consistency null
2022-03-22 06:32:25.601 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [53229130-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:26.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789148480-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3951d81f at consistency null
2022-03-22 06:32:26.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [789148480-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:26.012 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:32:31.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2019350077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@373d03d1 at consistency null
2022-03-22 06:32:31.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2019350077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:31.202 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1204023180-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6399b814 at consistency null
2022-03-22 06:32:31.206 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1204023180-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:31.335 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633282392-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b96ace0 at consistency null
2022-03-22 06:32:31.335 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [633282392-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:31.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1103686620-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c3012ad at consistency null
2022-03-22 06:32:31.443 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1103686620-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:31.568 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1597714662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@134cb371 at consistency null
2022-03-22 06:32:31.569 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1597714662-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:31.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [131861250-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d3bf014 at consistency null
2022-03-22 06:32:31.712 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [131861250-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:31.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1522831742-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7304aaff at consistency null
2022-03-22 06:32:31.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1522831742-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:32.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549010929-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f6d6b48 at consistency null
2022-03-22 06:32:32.211 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1549010929-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:32.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1517203613-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49cc80ef at consistency null
2022-03-22 06:32:32.425 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1517203613-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:32.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1024275657-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b956292 at consistency null
2022-03-22 06:32:32.639 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1024275657-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:33.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1128013175-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d10764d at consistency null
2022-03-22 06:32:33.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1128013175-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:33.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1068233070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2973658c at consistency null
2022-03-22 06:32:33.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1068233070-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:34.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284054622-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49553b5 at consistency null
2022-03-22 06:32:34.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [284054622-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:34.315 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:32:39.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1141998070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4142047 at consistency null
2022-03-22 06:32:39.336 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1141998070-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:39.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1734736806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a489c96 at consistency null
2022-03-22 06:32:39.402 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1734736806-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:39.457 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [128272037-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48a5d956 at consistency null
2022-03-22 06:32:39.458 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [128272037-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:39.579 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703653035-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6603230c at consistency null
2022-03-22 06:32:39.580 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703653035-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:39.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [761105885-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43b03193 at consistency null
2022-03-22 06:32:39.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [761105885-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:40.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1697508123-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7da2b95f at consistency null
2022-03-22 06:32:40.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1697508123-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:41.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1217403253-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1baba0d4 at consistency null
2022-03-22 06:32:41.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1217403253-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:42.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1884690490-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@644febee at consistency null
2022-03-22 06:32:42.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1884690490-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:42.639 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:32:43.629 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:32:43.643 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:32:47.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [824459400-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26b1725d at consistency null
2022-03-22 06:32:47.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [824459400-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:47.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [995819652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e314f95 at consistency null
2022-03-22 06:32:47.803 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [995819652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:47.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1942547755-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@85aeb87 at consistency null
2022-03-22 06:32:47.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1942547755-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:48.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866102432-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@135171f9 at consistency null
2022-03-22 06:32:48.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [866102432-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:48.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [548178523-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e5fda54 at consistency null
2022-03-22 06:32:48.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [548178523-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:48.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1125183978-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ad667ed at consistency null
2022-03-22 06:32:48.418 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1125183978-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:48.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1996881174-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@258fbde3 at consistency null
2022-03-22 06:32:48.704 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1996881174-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:49.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [959555066-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d49de12 at consistency null
2022-03-22 06:32:49.208 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [959555066-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:49.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2106269397-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d842e95 at consistency null
2022-03-22 06:32:49.621 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2106269397-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:50.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961560367-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34a4c7c7 at consistency null
2022-03-22 06:32:50.516 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [961560367-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:50.519 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:32:55.562 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1642432606-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c9c1415 at consistency null
2022-03-22 06:32:55.562 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1642432606-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:55.616 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1439694829-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f56feda at consistency null
2022-03-22 06:32:55.617 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1439694829-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:55.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1880567351-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43fa9b23 at consistency null
2022-03-22 06:32:55.804 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1880567351-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:56.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1216794205-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51f2e664 at consistency null
2022-03-22 06:32:56.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1216794205-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:56.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [99358827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dc26cf3 at consistency null
2022-03-22 06:32:56.533 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [99358827-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:57.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [796307372-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@754ae9bf at consistency null
2022-03-22 06:32:57.011 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [796307372-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:57.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1725520611-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@627e47b6 at consistency null
2022-03-22 06:32:57.824 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1725520611-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:58.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [408130627-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24794c5b at consistency null
2022-03-22 06:32:58.705 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [408130627-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:32:58.707 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:33:03.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2107238498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a9209d3 at consistency null
2022-03-22 06:33:03.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2107238498-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:03.820 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728586674-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48b7022f at consistency null
2022-03-22 06:33:03.821 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1728586674-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:03.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261073394-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36112bbb at consistency null
2022-03-22 06:33:03.925 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1261073394-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:04.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340020568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cb7603b at consistency null
2022-03-22 06:33:04.032 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1340020568-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:04.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2109156409-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c61b5f8 at consistency null
2022-03-22 06:33:04.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2109156409-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:04.434 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1527022990-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4103c263 at consistency null
2022-03-22 06:33:04.435 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1527022990-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:04.901 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1126811903-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ef0750b at consistency null
2022-03-22 06:33:04.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1126811903-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:05.305 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1954247189-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9b16456 at consistency null
2022-03-22 06:33:05.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1954247189-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:05.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [812207687-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@384edeed at consistency null
2022-03-22 06:33:05.722 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [812207687-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:06.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1337886001-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6701135d at consistency null
2022-03-22 06:33:06.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1337886001-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:07.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1121833481-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a246a9b at consistency null
2022-03-22 06:33:07.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1121833481-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:07.336 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:33:12.342 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1838803869-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@18352e5d at consistency null
2022-03-22 06:33:12.343 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1838803869-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:12.370 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [393779999-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56c23200 at consistency null
2022-03-22 06:33:12.370 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [393779999-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:12.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743703500-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3936487e at consistency null
2022-03-22 06:33:12.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [743703500-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:12.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [11911637-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f9d71 at consistency null
2022-03-22 06:33:12.543 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [11911637-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:12.653 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575801181-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31090ce5 at consistency null
2022-03-22 06:33:12.653 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [575801181-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:12.857 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1898037355-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@769c7af6 at consistency null
2022-03-22 06:33:12.858 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1898037355-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:13.263 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103965664-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7de2b9e3 at consistency null
2022-03-22 06:33:13.263 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2103965664-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:13.659 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:33:13.662 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:33:14.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399995120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10a373b at consistency null
2022-03-22 06:33:14.070 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399995120-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:15.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1638446892-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@789d7ddb at consistency null
2022-03-22 06:33:15.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1638446892-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:15.697 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:33:20.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510988778-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d43e3f4 at consistency null
2022-03-22 06:33:20.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [510988778-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:20.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550794035-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@506621e at consistency null
2022-03-22 06:33:20.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1550794035-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:20.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [503855604-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55e53e6a at consistency null
2022-03-22 06:33:20.905 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [503855604-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:21.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985529550-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4685e2a8 at consistency null
2022-03-22 06:33:21.022 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985529550-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:21.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [217141298-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34abf5cf at consistency null
2022-03-22 06:33:21.318 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [217141298-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:21.739 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [701657658-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28b77d6 at consistency null
2022-03-22 06:33:21.739 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [701657658-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:22.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1984575853-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ee10f83 at consistency null
2022-03-22 06:33:22.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1984575853-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:23.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [591720678-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17183255 at consistency null
2022-03-22 06:33:23.504 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [591720678-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:24.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632444028-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c76b8f4 at consistency null
2022-03-22 06:33:24.319 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [632444028-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:24.320 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:33:29.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533442898-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@251a456b at consistency null
2022-03-22 06:33:29.404 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [533442898-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:29.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [955081367-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53f1cb5c at consistency null
2022-03-22 06:33:29.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [955081367-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:29.628 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1017804460-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cd568b4 at consistency null
2022-03-22 06:33:29.629 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1017804460-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:29.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [185390321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b134d58 at consistency null
2022-03-22 06:33:29.911 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [185390321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:30.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1225762275-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6665da4d at consistency null
2022-03-22 06:33:30.118 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1225762275-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:30.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [438673295-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@dac4dd5 at consistency null
2022-03-22 06:33:30.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [438673295-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:31.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1968543089-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6adb041c at consistency null
2022-03-22 06:33:31.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1968543089-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:31.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [403926134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b83fd92 at consistency null
2022-03-22 06:33:31.822 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [403926134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:32.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [747102762-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37cf11d at consistency null
2022-03-22 06:33:32.708 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [747102762-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:32.711 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:33:37.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [443079122-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c51d642 at consistency null
2022-03-22 06:33:37.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [443079122-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:37.773 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129265651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e18d430 at consistency null
2022-03-22 06:33:37.773 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1129265651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:37.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1077710128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a23dd0c at consistency null
2022-03-22 06:33:37.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1077710128-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:37.986 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1404678575-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e33314c at consistency null
2022-03-22 06:33:37.992 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1404678575-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:38.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256066627-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cb2090e at consistency null
2022-03-22 06:33:38.101 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [256066627-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:38.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897152249-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1493f5fc at consistency null
2022-03-22 06:33:38.224 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [897152249-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:38.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [285747819-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c81df at consistency null
2022-03-22 06:33:38.429 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [285747819-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:38.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [253941036-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@778269cb at consistency null
2022-03-22 06:33:38.833 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [253941036-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:39.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [450292183-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b64e44c at consistency null
2022-03-22 06:33:39.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [450292183-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:41.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693882568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60d3c0c9 at consistency null
2022-03-22 06:33:41.242 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1693882568-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:41.244 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:33:43.699 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:33:43.704 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:33:46.249 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889044303-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f5e775 at consistency null
2022-03-22 06:33:46.249 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [889044303-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:46.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [762283337-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@605d0a11 at consistency null
2022-03-22 06:33:46.303 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [762283337-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:46.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [14305634-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c57d4d2 at consistency null
2022-03-22 06:33:46.362 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [14305634-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:46.423 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1346019689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52fc9c19 at consistency null
2022-03-22 06:33:46.424 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1346019689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:46.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [167012836-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7919077c at consistency null
2022-03-22 06:33:46.530 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [167012836-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:46.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1491132510-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@711c97cc at consistency null
2022-03-22 06:33:46.637 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1491132510-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:46.850 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [178463609-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63ff3c79 at consistency null
2022-03-22 06:33:46.851 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [178463609-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:47.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [261614253-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55d5684b at consistency null
2022-03-22 06:33:47.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [261614253-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:47.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629541362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64c6b76a at consistency null
2022-03-22 06:33:47.314 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [629541362-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:47.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2050752268-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7171fec9 at consistency null
2022-03-22 06:33:47.732 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2050752268-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:48.140 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1762698201-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b77086a at consistency null
2022-03-22 06:33:48.140 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1762698201-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:48.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1508048596-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@159eb27f at consistency null
2022-03-22 06:33:48.947 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1508048596-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:49.785 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1279178115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44323a41 at consistency null
2022-03-22 06:33:49.785 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1279178115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:49.787 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:33:54.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1767436439-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3acb03b8 at consistency null
2022-03-22 06:33:54.817 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1767436439-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:54.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [87726094-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69ac68d3 at consistency null
2022-03-22 06:33:54.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [87726094-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:55.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273025059-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9c122d3 at consistency null
2022-03-22 06:33:55.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1273025059-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:55.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1917580401-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cf4f198 at consistency null
2022-03-22 06:33:55.112 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1917580401-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:55.225 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1547271911-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e40ff7d at consistency null
2022-03-22 06:33:55.226 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1547271911-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:55.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [219309958-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34a0f296 at consistency null
2022-03-22 06:33:55.528 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [219309958-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:55.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1085197773-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3477896b at consistency null
2022-03-22 06:33:55.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1085197773-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:56.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1844138453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f74d64c at consistency null
2022-03-22 06:33:56.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1844138453-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:56.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2058536137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@767663c8 at consistency null
2022-03-22 06:33:56.915 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2058536137-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:57.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1686286926-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45f11bd2 at consistency null
2022-03-22 06:33:57.731 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1686286926-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:33:57.733 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:435)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:34:02.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383390134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dc5eed6 at consistency null
2022-03-22 06:34:02.812 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1383390134-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:02.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1705397716-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@246a8794 at consistency null
2022-03-22 06:34:02.920 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1705397716-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:02.991 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165562661-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61c7d8f2 at consistency null
2022-03-22 06:34:02.991 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [165562661-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:03.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025822866-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ca57093 at consistency null
2022-03-22 06:34:03.115 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2025822866-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:03.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [216160088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ef4988f at consistency null
2022-03-22 06:34:03.310 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [216160088-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:03.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1543332861-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e52f0e2 at consistency null
2022-03-22 06:34:03.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1543332861-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:03.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [546414278-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@610dbf92 at consistency null
2022-03-22 06:34:03.825 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [546414278-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:04.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [612297548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@662b7ac8 at consistency null
2022-03-22 06:34:04.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [612297548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:04.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985145356-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c71a662 at consistency null
2022-03-22 06:34:04.534 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [985145356-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:05.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [183131693-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@250d92ee at consistency null
2022-03-22 06:34:05.338 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [183131693-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:05.340 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:34:10.360 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [648123928-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29f058f2 at consistency null
2022-03-22 06:34:10.360 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [648123928-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:10.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1968585995-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14308a3e at consistency null
2022-03-22 06:34:10.426 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1968585995-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:10.482 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [244702189-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15f36c6c at consistency null
2022-03-22 06:34:10.482 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [244702189-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:10.593 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1456924452-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5ca016de at consistency null
2022-03-22 06:34:10.596 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1456924452-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:10.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095954608-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77bcf270 at consistency null
2022-03-22 06:34:10.709 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2095954608-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:10.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941063747-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67cdf7c3 at consistency null
2022-03-22 06:34:10.929 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [941063747-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:11.144 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [410882079-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69d3775a at consistency null
2022-03-22 06:34:11.145 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [410882079-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:11.589 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046012403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35198a58 at consistency null
2022-03-22 06:34:11.589 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2046012403-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:12.399 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1750775367-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2526c1f9 at consistency null
2022-03-22 06:34:12.399 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1750775367-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:13.721 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:34:13.726 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:34:14.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1807497321-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ddd4971 at consistency null
2022-03-22 06:34:14.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1807497321-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:14.029 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:34:19.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523608169-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f468237 at consistency null
2022-03-22 06:34:19.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [523608169-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:19.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2080193615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@267498cd at consistency null
2022-03-22 06:34:19.229 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2080193615-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:19.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1185437177-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35ceacf3 at consistency null
2022-03-22 06:34:19.333 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1185437177-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:19.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281180213-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@729ffdfa at consistency null
2022-03-22 06:34:19.608 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1281180213-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:20.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128188735-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b93cfbd at consistency null
2022-03-22 06:34:20.025 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2128188735-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:20.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1817498095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b7a5108 at consistency null
2022-03-22 06:34:20.914 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1817498095-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:21.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1758938625-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@560707b0 at consistency null
2022-03-22 06:34:21.826 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1758938625-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:21.829 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:34:26.912 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1312685508-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5354f57a at consistency null
2022-03-22 06:34:26.913 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1312685508-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:27.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088757585-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@563f368a at consistency null
2022-03-22 06:34:27.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2088757585-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:27.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [913216664-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f44cf54 at consistency null
2022-03-22 06:34:27.123 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [913216664-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:27.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692085018-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3110798b at consistency null
2022-03-22 06:34:27.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1692085018-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:27.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1168718232-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@451ba5f6 at consistency null
2022-03-22 06:34:27.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1168718232-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:27.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176462621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6abf50b9 at consistency null
2022-03-22 06:34:27.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1176462621-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:27.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68674322-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13f81cd5 at consistency null
2022-03-22 06:34:27.707 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68674322-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:27.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [967416689-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2196f300 at consistency null
2022-03-22 06:34:27.927 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [967416689-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:28.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [901775397-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39bad6d1 at consistency null
2022-03-22 06:34:28.352 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [901775397-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:29.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147504141-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@299073c7 at consistency null
2022-03-22 06:34:29.157 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [147504141-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:30.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [863945077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7adddc94 at consistency null
2022-03-22 06:34:30.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [863945077-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:30.012 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:34:35.103 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1182381688-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c452e3c at consistency null
2022-03-22 06:34:35.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1182381688-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:35.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1938638778-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a6613b at consistency null
2022-03-22 06:34:35.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1938638778-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:35.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [788178706-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@290224c3 at consistency null
2022-03-22 06:34:35.323 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [788178706-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:35.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1808445137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3daf740b at consistency null
2022-03-22 06:34:35.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1808445137-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:36.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611337638-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@785e894c at consistency null
2022-03-22 06:34:36.027 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1611337638-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:36.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2032284810-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48c180c5 at consistency null
2022-03-22 06:34:36.831 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2032284810-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:37.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43765387-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@705d8122 at consistency null
2022-03-22 06:34:37.636 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [43765387-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:38.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1016681519-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3614ac0e at consistency null
2022-03-22 06:34:38.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1016681519-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:38.444 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:34:43.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72813093-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3a0c5769 at consistency null
2022-03-22 06:34:43.451 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [72813093-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:43.482 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1824122945-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e0f3f45 at consistency null
2022-03-22 06:34:43.482 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1824122945-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:43.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [814390475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62c7e537 at consistency null
2022-03-22 06:34:43.537 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [814390475-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:43.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1187290841-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@712041bd at consistency null
2022-03-22 06:34:43.642 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1187290841-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:43.746 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2055119366-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29cff7ef at consistency null
2022-03-22 06:34:43.747 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2055119366-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:43.767 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:34:43.773 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:34:43.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495709416-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ebd0084 at consistency null
2022-03-22 06:34:43.958 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1495709416-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:44.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1034814915-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@730eb639 at consistency null
2022-03-22 06:34:44.163 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1034814915-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:44.389 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1602119417-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d03c4c at consistency null
2022-03-22 06:34:44.390 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1602119417-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:44.796 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1352630658-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5676d336 at consistency null
2022-03-22 06:34:44.796 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1352630658-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:45.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1021504655-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1009405c at consistency null
2022-03-22 06:34:45.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1021504655-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:47.217 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1898931917-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16b0931a at consistency null
2022-03-22 06:34:47.218 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1898931917-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:47.221 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:34:52.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1565171647-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1783efdc at consistency null
2022-03-22 06:34:52.312 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1565171647-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:52.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903527429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@778c7179 at consistency null
2022-03-22 06:34:52.436 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1903527429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:52.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1515756080-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d5dd474 at consistency null
2022-03-22 06:34:52.518 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1515756080-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:52.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355519672-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f67aa86 at consistency null
2022-03-22 06:34:52.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1355519672-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:52.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1509248121-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37124fb9 at consistency null
2022-03-22 06:34:52.721 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1509248121-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:52.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74300652-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60bc79fb at consistency null
2022-03-22 06:34:52.818 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [74300652-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:52.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [452603723-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@620ddcf0 at consistency null
2022-03-22 06:34:52.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [452603723-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:53.124 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [378012922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47c7288f at consistency null
2022-03-22 06:34:53.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [378012922-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:53.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1701647574-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b56b2c3 at consistency null
2022-03-22 06:34:53.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1701647574-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:53.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [210652747-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68a66df3 at consistency null
2022-03-22 06:34:53.910 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [210652747-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:54.416 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1204140262-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64c0c64b at consistency null
2022-03-22 06:34:54.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1204140262-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:54.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [289200944-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a3d190c at consistency null
2022-03-22 06:34:54.933 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [289200944-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:55.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486188645-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44237022 at consistency null
2022-03-22 06:34:55.827 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [486188645-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:34:55.829 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:35:00.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1652055975-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f3154a0 at consistency null
2022-03-22 06:35:00.903 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1652055975-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:01.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [143706502-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7da13b93 at consistency null
2022-03-22 06:35:01.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [143706502-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:01.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2130867967-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fa6b852 at consistency null
2022-03-22 06:35:01.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2130867967-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:01.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914505460-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50db6be4 at consistency null
2022-03-22 06:35:01.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [914505460-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:01.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371710692-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45daa10b at consistency null
2022-03-22 06:35:01.716 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [371710692-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:01.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [410385759-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31d36780 at consistency null
2022-03-22 06:35:01.936 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [410385759-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:02.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957483808-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b0d3dc0 at consistency null
2022-03-22 06:35:02.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1957483808-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:03.168 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1409163487-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f30e at consistency null
2022-03-22 06:35:03.168 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1409163487-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:04.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066100414-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10c0c8fc at consistency null
2022-03-22 06:35:04.806 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2066100414-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:04.807 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:35:06.820 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] preparing to open 1 new connections, total = 1
2022-03-22 06:35:06.851 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-49, inFlight=0, closed=false] Error connecting to /10.233.71.64:9042 (Connection refused: /10.233.71.64:9042)
2022-03-22 06:35:06.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.71.64:9042-49, inFlight=0, closed=false] because: [/10.233.71.64:9042] Cannot connect
2022-03-22 06:35:06.852 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] Connection[/10.233.71.64:9042-49, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 06:35:06.854 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.233.71.64:9042-49, inFlight=0, closed=true] closing connection
2022-03-22 06:35:06.855 DEBUG 10 --- [uster-worker-89] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.71.64:9042
2022-03-22 06:35:06.856 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.71.64:9042 ([/10.233.71.64:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 06:35:06.860 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.233.71.64:9042] next reconnection attempt in 600000 ms
2022-03-22 06:35:09.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1284420992-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59983f06 at consistency null
2022-03-22 06:35:09.837 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1284420992-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:09.890 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578728676-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@499dfb7c at consistency null
2022-03-22 06:35:09.890 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1578728676-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:09.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2093176608-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3538c09a at consistency null
2022-03-22 06:35:09.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2093176608-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:10.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917126086-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@367b179f at consistency null
2022-03-22 06:35:10.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917126086-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:10.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254917158-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e036e59 at consistency null
2022-03-22 06:35:10.277 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [254917158-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:10.692 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [442112376-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8e326fe at consistency null
2022-03-22 06:35:10.692 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [442112376-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:11.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2084972356-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bac0b68 at consistency null
2022-03-22 06:35:11.509 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2084972356-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:13.134 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466962177-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15052c6d at consistency null
2022-03-22 06:35:13.134 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [466962177-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:13.135 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:35:13.797 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:35:13.837 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:35:18.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431435479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20c1f932 at consistency null
2022-03-22 06:35:18.204 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [431435479-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:18.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399616832-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3104ef02 at consistency null
2022-03-22 06:35:18.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1399616832-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:18.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [517805166-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ae84501 at consistency null
2022-03-22 06:35:18.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [517805166-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:18.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [797029883-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16b27275 at consistency null
2022-03-22 06:35:18.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [797029883-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:18.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258013002-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@391501f8 at consistency null
2022-03-22 06:35:18.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [258013002-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:18.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1058428307-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fff8527 at consistency null
2022-03-22 06:35:18.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1058428307-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:18.828 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235110011-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47d71d94 at consistency null
2022-03-22 06:35:18.829 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1235110011-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:19.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1796346590-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42f3eec1 at consistency null
2022-03-22 06:35:19.133 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1796346590-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:19.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1665163857-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@53783da6 at consistency null
2022-03-22 06:35:19.542 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1665163857-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:20.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1853294951-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fd4c53 at consistency null
2022-03-22 06:35:20.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1853294951-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:22.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1435584834-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f39da65 at consistency null
2022-03-22 06:35:22.028 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1435584834-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:22.030 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:35:27.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [787359180-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2261b6a5 at consistency null
2022-03-22 06:35:27.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [787359180-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:27.507 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1314687146-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30e63a42 at consistency null
2022-03-22 06:35:27.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1314687146-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:27.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [414616631-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a1367d9 at consistency null
2022-03-22 06:35:27.611 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [414616631-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:27.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202855226-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b48640c at consistency null
2022-03-22 06:35:27.714 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1202855226-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:27.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [414027845-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4433235e at consistency null
2022-03-22 06:35:27.839 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [414027845-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:27.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [840843810-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@458625b0 at consistency null
2022-03-22 06:35:27.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [840843810-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:28.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [518367074-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4eaaf76 at consistency null
2022-03-22 06:35:28.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [518367074-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:28.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1231630921-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2caac3e3 at consistency null
2022-03-22 06:35:28.209 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1231630921-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:28.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1035123865-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71de3cfb at consistency null
2022-03-22 06:35:28.331 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1035123865-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:28.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2105402975-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5013cd5 at consistency null
2022-03-22 06:35:28.409 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2105402975-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:28.524 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318096933-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54c1b21c at consistency null
2022-03-22 06:35:28.525 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1318096933-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:28.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324026052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e0ed6bc at consistency null
2022-03-22 06:35:28.819 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324026052-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:29.334 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1712661492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2eb70e62 at consistency null
2022-03-22 06:35:29.334 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1712661492-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:29.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950903927-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@225c68f5 at consistency null
2022-03-22 06:35:29.815 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1950903927-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:30.307 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1041488227-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b547ff0 at consistency null
2022-03-22 06:35:30.308 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1041488227-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:30.319 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:35:35.367 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [682117965-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23b7ed10 at consistency null
2022-03-22 06:35:35.367 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [682117965-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:35.490 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1743640367-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3565231a at consistency null
2022-03-22 06:35:35.490 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1743640367-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:35.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348544115-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@743fee83 at consistency null
2022-03-22 06:35:35.634 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1348544115-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:35.757 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1889088989-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ccca40b at consistency null
2022-03-22 06:35:35.758 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1889088989-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:35.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [444721824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@300f03ef at consistency null
2022-03-22 06:35:35.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [444721824-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:36.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [324797040-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2cee9b07 at consistency null
2022-03-22 06:35:36.040 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [324797040-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:36.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1247276246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27ae23ef at consistency null
2022-03-22 06:35:36.223 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1247276246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:36.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1943605148-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e8b37ed at consistency null
2022-03-22 06:35:36.406 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1943605148-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:36.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721249709-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@265341a8 at consistency null
2022-03-22 06:35:36.641 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1721249709-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:37.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [354741900-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e3dd0c6 at consistency null
2022-03-22 06:35:37.110 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [354741900-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:37.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1625330082-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28da35b5 at consistency null
2022-03-22 06:35:37.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1625330082-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:38.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535039437-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f595b1c at consistency null
2022-03-22 06:35:38.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1535039437-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:38.525 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:35:40.609 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] preparing to open 1 new connections, total = 1
2022-03-22 06:35:40.640 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-73, inFlight=0, closed=false] Error connecting to /10.233.66.75:9042 (Connection refused: /10.233.66.75:9042)
2022-03-22 06:35:40.640 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.233.66.75:9042-73, inFlight=0, closed=false] because: [/10.233.66.75:9042] Cannot connect
2022-03-22 06:35:40.640 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] Connection[/10.233.66.75:9042-73, inFlight=0, closed=false] failed, remaining = 0
2022-03-22 06:35:40.641 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.233.66.75:9042-73, inFlight=0, closed=true] closing connection
2022-03-22 06:35:40.642 DEBUG 10 --- [uster-worker-90] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.233.66.75:9042
2022-03-22 06:35:40.642 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.233.66.75:9042 ([/10.233.66.75:9042] Cannot connect), scheduling retry in 600000 milliseconds
2022-03-22 06:35:40.643 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.233.66.75:9042] next reconnection attempt in 600000 ms
2022-03-22 06:35:43.563 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2084339272-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@23bb175e at consistency null
2022-03-22 06:35:43.564 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2084339272-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:43.633 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1323220941-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3390480f at consistency null
2022-03-22 06:35:43.633 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1323220941-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:43.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718756405-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a1eb455 at consistency null
2022-03-22 06:35:43.719 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [718756405-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:43.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129638429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fca87b1 at consistency null
2022-03-22 06:35:43.809 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [129638429-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:43.838 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:35:43.874 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:35:43.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [409714844-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@643062f1 at consistency null
2022-03-22 06:35:43.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [409714844-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:44.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [602940555-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55d19e48 at consistency null
2022-03-22 06:35:44.057 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [602940555-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:44.291 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862377010-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ad224bb at consistency null
2022-03-22 06:35:44.291 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [862377010-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:44.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703587979-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@216ed15f at consistency null
2022-03-22 06:35:44.703 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1703587979-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:45.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589554566-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@462e1869 at consistency null
2022-03-22 06:35:45.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [589554566-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:46.339 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [22368021-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f54ed3a at consistency null
2022-03-22 06:35:46.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [22368021-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:46.341 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:35:51.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68672585-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6458e597 at consistency null
2022-03-22 06:35:51.430 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [68672585-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:51.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [883505832-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33eb346a at consistency null
2022-03-22 06:35:51.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [883505832-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:51.613 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [145408104-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5309f2e at consistency null
2022-03-22 06:35:51.614 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [145408104-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:51.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598481807-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3db58f8f at consistency null
2022-03-22 06:35:51.718 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598481807-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:51.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [434335271-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ee2a1f3 at consistency null
2022-03-22 06:35:51.844 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [434335271-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:52.130 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1684171721-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74408e21 at consistency null
2022-03-22 06:35:52.131 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1684171721-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:52.387 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [974479021-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64d096a1 at consistency null
2022-03-22 06:35:52.387 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [974479021-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:52.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040941059-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f364ef1 at consistency null
2022-03-22 06:35:52.814 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2040941059-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:53.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2067173702-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ff7f4b0 at consistency null
2022-03-22 06:35:53.618 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2067173702-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:55.327 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [217944522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f7c2ad0 at consistency null
2022-03-22 06:35:55.329 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [217944522-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:35:55.331 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:36:00.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479020750-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ff9d400 at consistency null
2022-03-22 06:36:00.442 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [479020750-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:00.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1471213838-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12654ac8 at consistency null
2022-03-22 06:36:00.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1471213838-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:00.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407103007-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32176e40 at consistency null
2022-03-22 06:36:00.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [407103007-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:00.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026919708-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cf3e878 at consistency null
2022-03-22 06:36:00.711 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2026919708-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:00.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245220862-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dab7a32 at consistency null
2022-03-22 06:36:00.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [245220862-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:00.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1584758178-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45a05c45 at consistency null
2022-03-22 06:36:00.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1584758178-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:00.963 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1493240685-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4c4c4fdf at consistency null
2022-03-22 06:36:00.963 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1493240685-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:01.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [588500966-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f9fde06 at consistency null
2022-03-22 06:36:01.104 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [588500966-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:01.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1128356943-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9434b2d at consistency null
2022-03-22 06:36:01.321 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1128356943-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:01.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917283304-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4ec66c34 at consistency null
2022-03-22 06:36:01.610 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [917283304-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:02.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1730649651-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60dae084 at consistency null
2022-03-22 06:36:02.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1730649651-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:02.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1517687246-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31646d77 at consistency null
2022-03-22 06:36:02.924 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1517687246-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:03.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [329027051-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22ed4255 at consistency null
2022-03-22 06:36:03.832 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [329027051-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:03.834 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:36:08.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1080876433-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f372df0 at consistency null
2022-03-22 06:36:08.842 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1080876433-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:08.895 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [459347369-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@685e3852 at consistency null
2022-03-22 06:36:08.896 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [459347369-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:08.950 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1911787370-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ac3fffb at consistency null
2022-03-22 06:36:08.951 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1911787370-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:09.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [108504938-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3274e51c at consistency null
2022-03-22 06:36:09.033 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [108504938-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:09.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542208608-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54ca160f at consistency null
2022-03-22 06:36:09.106 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [542208608-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:09.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [865007518-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d4f386c at consistency null
2022-03-22 06:36:09.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [865007518-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:09.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1592505946-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28163a19 at consistency null
2022-03-22 06:36:09.344 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1592505946-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:09.457 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1696335386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3ddf7602 at consistency null
2022-03-22 06:36:09.458 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1696335386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:09.674 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324722859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@392165d8 at consistency null
2022-03-22 06:36:09.674 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1324722859-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:10.089 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [168003308-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d19a6c8 at consistency null
2022-03-22 06:36:10.089 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [168003308-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:10.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156874896-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14c69380 at consistency null
2022-03-22 06:36:10.506 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [156874896-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:11.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208683895-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@526f987c at consistency null
2022-03-22 06:36:11.340 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [208683895-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:12.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1874006176-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60d2284c at consistency null
2022-03-22 06:36:12.146 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1874006176-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:12.148 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:36:13.875 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2022-03-22 06:36:13.899 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.233.68.68:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2022-03-22 06:36:17.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2039598940-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@203ded78 at consistency null
2022-03-22 06:36:17.207 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2039598940-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:17.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1036543430-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47306c at consistency null
2022-03-22 06:36:17.306 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1036543430-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:17.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [858645235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e45f460 at consistency null
2022-03-22 06:36:17.414 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [858645235-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:17.630 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1156366886-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f2429ac at consistency null
2022-03-22 06:36:17.631 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1156366886-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:18.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [871374536-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@103d491d at consistency null
2022-03-22 06:36:18.102 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [871374536-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:19.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1307093797-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2940cf26 at consistency null
2022-03-22 06:36:19.023 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1307093797-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:19.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [294580753-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28a7d89a at consistency null
2022-03-22 06:36:19.935 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [294580753-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:19.936 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:36:25.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [869396468-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c3f3311 at consistency null
2022-03-22 06:36:25.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [869396468-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:25.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [6598936-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@329e04f at consistency null
2022-03-22 06:36:25.113 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [6598936-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:25.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [140350386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@155c9bb8 at consistency null
2022-03-22 06:36:25.252 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [140350386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:25.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1856370386-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44d6f7dd at consistency null
2022-03-22 06:36:25.508 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1856370386-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:25.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1387727549-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1124f3ed at consistency null
2022-03-22 06:36:25.638 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1387727549-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:25.918 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2013175170-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dc3521e at consistency null
2022-03-22 06:36:25.919 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2013175170-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:26.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [333070540-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c338345 at consistency null
2022-03-22 06:36:26.216 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [333070540-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:26.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1376270083-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f28f101 at consistency null
2022-03-22 06:36:26.702 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1376270083-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:27.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [939535645-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6925da92 at consistency null
2022-03-22 06:36:27.111 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [939535645-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:27.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [991016210-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e31360a at consistency null
2022-03-22 06:36:27.520 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [991016210-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:28.008 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2047626908-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ea8a122 at consistency null
2022-03-22 06:36:28.009 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2047626908-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:28.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1764740849-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@610fb0f at consistency null
2022-03-22 06:36:28.427 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1764740849-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:28.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1350462988-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41264015 at consistency null
2022-03-22 06:36:28.909 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1350462988-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:28.911 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2022-03-22 06:36:34.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1718745140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6680bf5d at consistency null
2022-03-22 06:36:34.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1718745140-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:34.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123444936-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d7ab31e at consistency null
2022-03-22 06:36:34.122 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [2123444936-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:34.214 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776165805-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fe62646 at consistency null
2022-03-22 06:36:34.215 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1776165805-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:34.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1249062548-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c7f12b0 at consistency null
2022-03-22 06:36:34.417 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1249062548-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:34.535 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598400950-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44f5b662 at consistency null
2022-03-22 06:36:34.536 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [598400950-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:34.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415233145-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@671cf3c6 at consistency null
2022-03-22 06:36:34.802 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1415233145-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:35.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1275149332-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19362d75 at consistency null
2022-03-22 06:36:35.018 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1275149332-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:35.606 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [525708430-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d085671 at consistency null
2022-03-22 06:36:35.607 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [525708430-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:36.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794607037-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24a7af14 at consistency null
2022-03-22 06:36:36.437 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.RequestHandler : [1794607037-0] Error querying cassandra/10.233.68.68:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2022-03-22 06:36:36.438 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2022-03-22T05:43:59.734Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/10.233.68.68:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted