By type
overriding logback.xml apexApps.sh: running application 'onappf' with command 'java -Dlogback.configurationFile=/opt/app/policy/apex-pdp/etc/logback.xml -cp /opt/app/policy/apex-pdp/etc:/opt/app/policy/apex-pdp/etc/hazelcast:/opt/app/policy/apex-pdp/etc/infinispan:/opt/app/policy/apex-pdp/lib/* -Djavax.net.ssl.keyStore=/opt/app/policy/apex-pdp/etc/ssl/policy-keystore -Djavax.net.ssl.keyStorePassword=Pol1cy_0nap -Djavax.net.ssl.trustStore=/opt/app/policy/apex-pdp/etc/ssl/policy-truststore -Djavax.net.ssl.trustStorePassword=Pol1cy_0nap -Dlogback.configurationFile=/opt/app/policy/apex-pdp/etc/logback.xml -Dhazelcast.config=/opt/app/policy/apex-pdp/etc/hazelcast.xml -Dhazelcast.mancenter.enabled=false org.onap.policy.apex.services.onappf.ApexStarterMain -c /home/apexuser/config/OnapPfConfig.json' [2022-09-07T23:13:43.363+00:00|INFO|ApexStarterMain|main] In ApexStarter with parameters [-c, /home/apexuser/config/OnapPfConfig.json] [2022-09-07T23:13:44.694+00:00|INFO|SingleThreadedDmaapTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: INITTED [2022-09-07T23:13:44.932+00:00|INFO|ServiceManager|main] service manager starting [2022-09-07T23:13:44.932+00:00|INFO|ServiceManager|main] service manager starting topics [2022-09-07T23:13:44.944+00:00|INFO|SingleThreadedBusTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: starting [2022-09-07T23:13:44.947+00:00|INFO|SingleThreadedDmaapTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: INITTED [2022-09-07T23:13:44.961+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:13:44.982+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:13:44.949+00:00|INFO|InlineBusTopicSink|main] InlineDmaapTopicSink [userName=null, password=null, getTopicCommInfrastructure()=DMAAP, toString()=InlineBusTopicSink [partitionId=e08ecfe8-6be7-4a48-85e9-39ec3c0eefbf, alive=false, publisher=null]]: starting [2022-09-07T23:13:45.019+00:00|INFO|InlineDmaapTopicSink|main] InlineDmaapTopicSink [userName=null, password=null, getTopicCommInfrastructure()=DMAAP, toString()=InlineBusTopicSink [partitionId=e08ecfe8-6be7-4a48-85e9-39ec3c0eefbf, alive=false, publisher=CambriaPublisherWrapper []]]: DMAAP SINK created [2022-09-07T23:13:45.019+00:00|INFO|ServiceManager|main] service manager starting set alive [2022-09-07T23:13:45.019+00:00|INFO|ServiceManager|main] service manager starting register pdp status context object [2022-09-07T23:13:45.028+00:00|INFO|ServiceManager|main] service manager starting topic sinks [2022-09-07T23:13:45.028+00:00|INFO|ServiceManager|main] service manager starting Pdp Status publisher [2022-09-07T23:13:45.165+00:00|INFO|ServiceManager|main] service manager starting Register pdp update listener [2022-09-07T23:13:45.170+00:00|INFO|ServiceManager|main] service manager starting Register pdp state change request dispatcher [2022-09-07T23:13:45.170+00:00|INFO|ServiceManager|main] service manager starting Message Dispatcher [2022-09-07T23:13:45.170+00:00|INFO|TopicBase|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: registering org.onap.policy.common.endpoints.listeners.MessageTypeDispatcher@7c7d3c46 [2022-09-07T23:13:45.172+00:00|INFO|SingleThreadedBusTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: register: start not attempted [2022-09-07T23:13:45.172+00:00|INFO|ServiceManager|main] service manager starting Create REST server [2022-09-07T23:13:45.173+00:00|WARN|PdpMessageHandler|Timer-0] not registered: object:engine/apex/handler [2022-09-07T23:13:45.235+00:00|INFO|OrderedServiceImpl|Timer-0] ***** OrderedServiceImpl implementers: [] [2022-09-07T23:13:45.240+00:00|INFO|network|Timer-0] [OUT|DMAAP|POLICY-PDP-PAP] {"pdpType":"apex","state":"PASSIVE","healthy":"HEALTHY","description":"Pdp Heartbeat","statistics":{"pdpInstanceId":"apex-59d63de8-902b-461a-ad49-275bc8a197f5","timeStamp":"2022-09-07T23:13:45.178570Z","pdpGroupName":"defaultGroup","policyExecutedCount":0,"policyExecutedSuccessCount":0,"policyExecutedFailCount":0,"policyDeployCount":0,"policyDeploySuccessCount":0,"policyDeployFailCount":0,"policyUndeployCount":0,"policyUndeploySuccessCount":0,"policyUndeployFailCount":0},"messageName":"PDP_STATUS","requestId":"16a8bd1a-65f8-4b64-a39a-228abb31e5b9","timestampMs":1662592425172,"name":"apex-59d63de8-902b-461a-ad49-275bc8a197f5","pdpGroup":"defaultGroup"} [2022-09-07T23:13:45.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 20 ms [2022-09-07T23:13:45.272+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:45.352+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:45.352+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:45.368+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:45.368+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:13:45.374+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:45.374+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:45.400+00:00|INFO|log|main] Logging initialized @3497ms to org.eclipse.jetty.util.log.Slf4jLog [2022-09-07T23:13:45.657+00:00|INFO|ServiceManager|main] service manager starting Rest Server [2022-09-07T23:13:45.657+00:00|INFO|ServiceManager|main] service manager starting [2022-09-07T23:13:45.657+00:00|INFO|ServiceManager|main] service manager starting REST RestServerParameters [2022-09-07T23:13:45.658+00:00|INFO|JettyServletServer|main] JettyJerseyServer [Jerseyservlets={/metrics=io.prometheus.client.exporter.MetricsServlet-74d7184a==io.prometheus.client.exporter.MetricsServlet@1bdaa749{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-3fabf088==org.glassfish.jersey.servlet.ServletContainer@49ac458c{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-82c57b3==io.swagger.jersey.config.JerseyJaxrsConfig@60916dc9{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}}, swaggerId=swagger-6969, toString()=JettyServletServer(name=RestServerParameters, host=0.0.0.0, port=6969, user=healthcheck, password=zb!XztG34, contextPath=/, jettyServer=Server@6c2f1700{STOPPED}[9.4.41.v20210516], context=o.e.j.s.ServletContextHandler@350b3a17{/,null,STOPPED}, connector=RestServerParameters@2ad3a1bb{HTTP/1.1, (http/1.1)}{0.0.0.0:6969}, jettyThread=null, servlets={/metrics=io.prometheus.client.exporter.MetricsServlet-74d7184a==io.prometheus.client.exporter.MetricsServlet@1bdaa749{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-3fabf088==org.glassfish.jersey.servlet.ServletContainer@49ac458c{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-82c57b3==io.swagger.jersey.config.JerseyJaxrsConfig@60916dc9{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}})]: STARTING [2022-09-07T23:13:45.703+00:00|INFO|ServiceManager|main] service manager started [2022-09-07T23:13:45.703+00:00|INFO|ServiceManager|main] service manager started [2022-09-07T23:13:45.704+00:00|INFO|ApexStarterMain|main] Started policy-apex-pdp service successfully. [2022-09-07T23:13:45.703+00:00|INFO|JettyServletServer|RestServerParameters-6969] JettyJerseyServer [Jerseyservlets={/metrics=io.prometheus.client.exporter.MetricsServlet-74d7184a==io.prometheus.client.exporter.MetricsServlet@1bdaa749{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-3fabf088==org.glassfish.jersey.servlet.ServletContainer@49ac458c{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-82c57b3==io.swagger.jersey.config.JerseyJaxrsConfig@60916dc9{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}}, swaggerId=swagger-6969, toString()=JettyServletServer(name=RestServerParameters, host=0.0.0.0, port=6969, user=healthcheck, password=zb!XztG34, contextPath=/, jettyServer=Server@6c2f1700{STOPPED}[9.4.41.v20210516], context=o.e.j.s.ServletContextHandler@350b3a17{/,null,STOPPED}, connector=RestServerParameters@2ad3a1bb{HTTP/1.1, (http/1.1)}{0.0.0.0:6969}, jettyThread=Thread[RestServerParameters-6969,5,main], servlets={/metrics=io.prometheus.client.exporter.MetricsServlet-74d7184a==io.prometheus.client.exporter.MetricsServlet@1bdaa749{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-3fabf088==org.glassfish.jersey.servlet.ServletContainer@49ac458c{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-82c57b3==io.swagger.jersey.config.JerseyJaxrsConfig@60916dc9{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}})]: STARTING [2022-09-07T23:13:45.712+00:00|INFO|Server|RestServerParameters-6969] jetty-9.4.41.v20210516; built: 2021-05-16T23:56:28.993Z; git: 98607f93c7833e7dc59489b13f3cb0a114fb9f4c; jvm 11.0.9+11-alpine-r1 [2022-09-07T23:13:45.789+00:00|INFO|session|RestServerParameters-6969] DefaultSessionIdManager workerName=node0 [2022-09-07T23:13:45.789+00:00|INFO|session|RestServerParameters-6969] No SessionScavenger set, using defaults [2022-09-07T23:13:45.792+00:00|INFO|session|RestServerParameters-6969] node0 Scavenging every 660000ms [2022-09-07T23:13:46.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 1170 ms [2022-09-07T23:13:46.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:46.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:46.423+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:46.423+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:46.424+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.sun.xml.bind.v2.runtime.reflect.opt.Injector (file:/opt/app/policy/apex-pdp/lib/jaxb-impl-2.3.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) WARNING: Please consider reporting this to the maintainers of com.sun.xml.bind.v2.runtime.reflect.opt.Injector WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [2022-09-07T23:13:47.315+00:00|INFO|YamlMessageBodyHandler|RestServerParameters-6969] Accepting YAML for REST calls [2022-09-07T23:13:47.315+00:00|INFO|GsonMessageBodyHandler|RestServerParameters-6969] Using GSON for REST calls [2022-09-07T23:13:47.473+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 2225 ms [2022-09-07T23:13:47.473+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:47.473+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:47.481+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:47.482+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:47.482+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:47.541+00:00|INFO|ContextHandler|RestServerParameters-6969] Started o.e.j.s.ServletContextHandler@350b3a17{/,null,AVAILABLE} [2022-09-07T23:13:47.574+00:00|INFO|AbstractConnector|RestServerParameters-6969] Started RestServerParameters@2ad3a1bb{HTTP/1.1, (http/1.1)}{0.0.0.0:6969} [2022-09-07T23:13:47.574+00:00|INFO|Server|RestServerParameters-6969] Started @5671ms [2022-09-07T23:13:48.523+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 3275 ms [2022-09-07T23:13:48.524+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:48.524+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:48.527+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:48.527+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:48.528+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:49.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 4321 ms [2022-09-07T23:13:49.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:49.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:49.574+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:49.575+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:49.576+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:50.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 5370 ms [2022-09-07T23:13:50.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:50.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:50.625+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:50.625+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:50.625+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:51.672+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 6422 ms [2022-09-07T23:13:51.672+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:51.673+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:51.679+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:51.680+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:51.681+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:52.719+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 7470 ms [2022-09-07T23:13:52.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:52.720+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:52.724+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:52.725+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:52.725+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:53.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 8521 ms [2022-09-07T23:13:53.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:53.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:53.772+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:53.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:53.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:54.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 9570 ms [2022-09-07T23:13:54.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:54.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:54.822+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:54.823+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:54.823+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:55.869+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 10621 ms [2022-09-07T23:13:55.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:55.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:55.873+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:55.873+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:55.875+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:55.875+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 11, above threshold 10. [2022-09-07T23:13:56.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 11670 ms [2022-09-07T23:13:56.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:56.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:56.924+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:56.925+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:56.925+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:56.925+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 12, above threshold 10. [2022-09-07T23:13:57.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 12720 ms [2022-09-07T23:13:57.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:57.970+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:57.973+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:57.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:57.974+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:57.974+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 13, above threshold 10. [2022-09-07T23:13:59.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 13771 ms [2022-09-07T23:13:59.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:13:59.020+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:13:59.023+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:13:59.024+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:13:59.024+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:13:59.024+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 14, above threshold 10. [2022-09-07T23:14:00.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 14820 ms [2022-09-07T23:14:00.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:00.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:00.072+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:00.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:00.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:00.073+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 15, above threshold 10. [2022-09-07T23:14:00.371+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:14:00.371+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:14:00.371+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:14:00.429+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:00.430+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:00.431+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:14:01.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 15870 ms [2022-09-07T23:14:01.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:01.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:01.122+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:01.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:01.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:01.122+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 16, above threshold 10. [2022-09-07T23:14:02.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 16920 ms [2022-09-07T23:14:02.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:02.170+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:02.175+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:02.176+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:02.176+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:02.176+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 17, above threshold 10. [2022-09-07T23:14:03.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 17970 ms [2022-09-07T23:14:03.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:03.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:03.222+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:03.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:03.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:03.222+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 18, above threshold 10. [2022-09-07T23:14:04.270+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 19021 ms [2022-09-07T23:14:04.270+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:04.270+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:04.274+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:04.274+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:04.274+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:04.274+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 19, above threshold 10. [2022-09-07T23:14:05.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 20071 ms [2022-09-07T23:14:05.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:05.320+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:05.323+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:05.323+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:05.323+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:05.323+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 20, above threshold 10. [2022-09-07T23:14:06.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 21120 ms [2022-09-07T23:14:06.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:06.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:06.372+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:06.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:06.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:06.373+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 21, above threshold 10. [2022-09-07T23:14:07.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 22171 ms [2022-09-07T23:14:07.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:07.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:07.425+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:07.426+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:07.427+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:07.427+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 22, above threshold 10. [2022-09-07T23:14:08.471+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 23223 ms [2022-09-07T23:14:08.471+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:08.472+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:08.475+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:08.476+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:08.476+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:08.476+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23, above threshold 10. [2022-09-07T23:14:09.616+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 24367 ms [2022-09-07T23:14:09.616+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:09.617+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:09.622+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:09.623+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:09.624+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:09.625+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 24, above threshold 10. [2022-09-07T23:14:10.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 25421 ms [2022-09-07T23:14:10.669+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:10.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:10.673+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:10.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:10.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:10.673+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 25, above threshold 10. [2022-09-07T23:14:11.719+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 26470 ms [2022-09-07T23:14:11.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:11.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:11.723+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:11.723+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:11.723+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:11.723+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 26, above threshold 10. [2022-09-07T23:14:12.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 27520 ms [2022-09-07T23:14:12.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:12.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:12.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:12.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:12.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:12.772+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 27, above threshold 10. [2022-09-07T23:14:13.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 28571 ms [2022-09-07T23:14:13.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:13.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:13.823+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:13.823+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:13.823+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:13.823+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 28, above threshold 10. [2022-09-07T23:14:14.873+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 29625 ms [2022-09-07T23:14:14.873+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:14.873+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:14.883+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:14.884+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:14.884+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:14.884+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 29, above threshold 10. [2022-09-07T23:14:15.432+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:14:15.432+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:14:15.433+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:14:15.443+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:15.443+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:15.444+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:14:15.921+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 30673 ms [2022-09-07T23:14:15.922+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:15.922+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:15.930+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:15.931+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:15.931+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:15.932+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 30, above threshold 10. [2022-09-07T23:14:16.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 31721 ms [2022-09-07T23:14:16.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:16.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:16.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:16.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:16.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:16.973+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 31, above threshold 10. [2022-09-07T23:14:18.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 32770 ms [2022-09-07T23:14:18.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:18.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:18.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:18.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:18.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:18.023+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 32, above threshold 10. [2022-09-07T23:14:19.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 33820 ms [2022-09-07T23:14:19.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:19.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:19.074+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:19.074+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:19.074+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:19.075+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 33, above threshold 10. [2022-09-07T23:14:20.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 34870 ms [2022-09-07T23:14:20.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:20.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:20.122+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:20.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:20.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:20.122+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 34, above threshold 10. [2022-09-07T23:14:21.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 35920 ms [2022-09-07T23:14:21.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:21.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:21.173+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:21.174+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:21.174+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:21.174+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 35, above threshold 10. [2022-09-07T23:14:22.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 36970 ms [2022-09-07T23:14:22.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:22.220+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:22.223+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:22.225+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:22.225+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:22.225+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 36, above threshold 10. [2022-09-07T23:14:23.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 38021 ms [2022-09-07T23:14:23.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:23.270+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:23.278+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:23.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:23.280+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:23.280+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 37, above threshold 10. [2022-09-07T23:14:24.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 39070 ms [2022-09-07T23:14:24.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:24.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:24.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:24.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:24.323+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:24.323+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 38, above threshold 10. [2022-09-07T23:14:25.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 40120 ms [2022-09-07T23:14:25.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:25.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:25.372+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:25.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:25.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:25.372+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 39, above threshold 10. [2022-09-07T23:14:26.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 41170 ms [2022-09-07T23:14:26.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:26.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:26.424+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:26.425+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:26.425+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:26.425+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 40, above threshold 10. [2022-09-07T23:14:27.469+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 42220 ms [2022-09-07T23:14:27.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:27.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:27.472+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:27.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:27.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:27.472+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 41, above threshold 10. [2022-09-07T23:14:28.519+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 43270 ms [2022-09-07T23:14:28.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:28.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:28.522+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:28.523+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:28.523+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:28.523+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 42, above threshold 10. [2022-09-07T23:14:29.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 44320 ms [2022-09-07T23:14:29.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:29.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:29.572+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:29.573+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:29.573+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:29.573+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 43, above threshold 10. [2022-09-07T23:14:30.444+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:14:30.444+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:14:30.445+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:14:30.449+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:30.449+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:30.449+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:14:30.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 45370 ms [2022-09-07T23:14:30.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:30.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:30.621+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:30.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:30.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:30.622+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 44, above threshold 10. [2022-09-07T23:14:31.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 46421 ms [2022-09-07T23:14:31.669+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:31.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:31.680+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:31.680+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:31.680+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:31.680+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 45, above threshold 10. [2022-09-07T23:14:32.719+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 47470 ms [2022-09-07T23:14:32.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:32.720+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:32.724+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:32.725+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:32.725+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:32.725+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 46, above threshold 10. [2022-09-07T23:14:33.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 48520 ms [2022-09-07T23:14:33.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:33.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:33.772+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:33.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:33.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:33.772+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 47, above threshold 10. [2022-09-07T23:14:34.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 49570 ms [2022-09-07T23:14:34.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:34.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:34.821+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:34.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:34.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:34.822+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 48, above threshold 10. [2022-09-07T23:14:35.869+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 50621 ms [2022-09-07T23:14:35.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:35.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:35.872+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:35.873+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:35.873+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:35.873+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 49, above threshold 10. [2022-09-07T23:14:36.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 51670 ms [2022-09-07T23:14:36.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:36.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:36.922+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:36.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:36.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:36.923+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 50, above threshold 10. [2022-09-07T23:14:37.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 52720 ms [2022-09-07T23:14:37.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:37.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:37.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:37.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:37.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:37.972+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 51, above threshold 10. [2022-09-07T23:14:39.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 53770 ms [2022-09-07T23:14:39.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:39.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:39.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:39.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:39.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:39.023+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 52, above threshold 10. [2022-09-07T23:14:40.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 54820 ms [2022-09-07T23:14:40.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:40.070+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:40.073+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:40.074+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:40.075+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:40.075+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 53, above threshold 10. [2022-09-07T23:14:41.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 55870 ms [2022-09-07T23:14:41.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:41.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:41.123+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:41.123+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:41.123+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:41.123+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 54, above threshold 10. [2022-09-07T23:14:42.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 56921 ms [2022-09-07T23:14:42.170+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:42.170+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:42.177+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:42.178+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:42.178+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:42.178+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 55, above threshold 10. [2022-09-07T23:14:43.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 57970 ms [2022-09-07T23:14:43.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:43.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:43.222+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:43.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:43.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:43.223+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 56, above threshold 10. [2022-09-07T23:14:44.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 59020 ms [2022-09-07T23:14:44.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:44.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:44.271+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:44.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:44.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:44.272+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 57, above threshold 10. [2022-09-07T23:14:45.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 60070 ms [2022-09-07T23:14:45.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:45.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:45.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:45.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:45.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:45.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 58, above threshold 10. [2022-09-07T23:14:45.450+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:14:45.450+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:14:45.450+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:14:45.454+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:45.454+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:45.454+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:14:46.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 61120 ms [2022-09-07T23:14:46.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:46.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:46.375+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:46.376+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:46.376+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:46.376+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 59, above threshold 10. [2022-09-07T23:14:47.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 62170 ms [2022-09-07T23:14:47.420+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:47.420+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:47.427+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:47.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:47.429+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:47.429+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 60, above threshold 10. [2022-09-07T23:14:48.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 63220 ms [2022-09-07T23:14:48.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:48.470+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:48.473+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:48.473+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:48.474+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:48.474+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 61, above threshold 10. [2022-09-07T23:14:49.519+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 64270 ms [2022-09-07T23:14:49.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:49.520+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:49.523+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:49.524+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:49.524+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:49.524+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 62, above threshold 10. [2022-09-07T23:14:50.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 65320 ms [2022-09-07T23:14:50.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:50.570+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:50.576+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:50.577+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:50.580+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:50.580+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 63, above threshold 10. [2022-09-07T23:14:51.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 66370 ms [2022-09-07T23:14:51.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:51.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:51.622+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:51.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:51.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:51.622+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 64, above threshold 10. [2022-09-07T23:14:52.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 67420 ms [2022-09-07T23:14:52.669+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:52.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:52.671+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:52.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:52.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:52.673+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 65, above threshold 10. [2022-09-07T23:14:53.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 68470 ms [2022-09-07T23:14:53.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:53.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:53.741+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:53.742+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:53.743+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:53.743+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 66, above threshold 10. [2022-09-07T23:14:54.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 69520 ms [2022-09-07T23:14:54.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:54.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:54.773+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:54.773+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:54.774+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:54.774+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 67, above threshold 10. [2022-09-07T23:14:55.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 70570 ms [2022-09-07T23:14:55.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:55.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:55.822+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:55.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:55.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:55.822+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 68, above threshold 10. [2022-09-07T23:14:56.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 71620 ms [2022-09-07T23:14:56.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:56.870+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:56.872+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:56.873+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:56.873+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:56.873+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 69, above threshold 10. [2022-09-07T23:14:57.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 72670 ms [2022-09-07T23:14:57.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:57.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:57.922+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:57.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:57.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:57.922+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 70, above threshold 10. [2022-09-07T23:14:58.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 73720 ms [2022-09-07T23:14:58.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:14:58.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:14:58.976+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:14:58.976+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:14:58.977+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:14:58.977+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 71, above threshold 10. [2022-09-07T23:15:00.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 74770 ms [2022-09-07T23:15:00.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:00.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:00.023+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:00.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:00.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:00.023+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 72, above threshold 10. [2022-09-07T23:15:00.457+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:15:00.477+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:15:00.477+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:15:00.485+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:00.485+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:00.485+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:15:01.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 75820 ms [2022-09-07T23:15:01.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:01.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:01.075+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:01.078+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:01.078+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:01.078+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 73, above threshold 10. [2022-09-07T23:15:02.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 76870 ms [2022-09-07T23:15:02.120+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:02.120+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:02.123+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:02.124+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:02.124+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:02.124+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 74, above threshold 10. [2022-09-07T23:15:03.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 77920 ms [2022-09-07T23:15:03.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:03.170+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:03.173+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:03.174+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:03.175+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:03.175+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 75, above threshold 10. [2022-09-07T23:15:04.220+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 78972 ms [2022-09-07T23:15:04.220+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:04.220+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:04.223+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:04.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:04.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:04.223+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 76, above threshold 10. [2022-09-07T23:15:05.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 80020 ms [2022-09-07T23:15:05.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:05.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:05.271+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:05.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:05.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:05.272+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 77, above threshold 10. [2022-09-07T23:15:06.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 81070 ms [2022-09-07T23:15:06.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:06.320+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:06.323+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:06.324+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:06.324+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:06.325+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 78, above threshold 10. [2022-09-07T23:15:07.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 82120 ms [2022-09-07T23:15:07.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:07.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:07.372+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:07.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:07.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:07.372+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 79, above threshold 10. [2022-09-07T23:15:08.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 83170 ms [2022-09-07T23:15:08.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:08.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:08.423+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:08.424+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:08.424+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:08.424+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 80, above threshold 10. [2022-09-07T23:15:09.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 84220 ms [2022-09-07T23:15:09.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:09.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:09.472+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:09.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:09.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:09.473+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 81, above threshold 10. [2022-09-07T23:15:10.521+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 85273 ms [2022-09-07T23:15:10.530+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:10.531+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:10.535+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:10.536+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:10.536+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:10.536+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 82, above threshold 10. [2022-09-07T23:15:11.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 86320 ms [2022-09-07T23:15:11.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:11.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:11.576+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:11.576+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:11.576+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:11.576+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 83, above threshold 10. [2022-09-07T23:15:12.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 87370 ms [2022-09-07T23:15:12.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:12.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:12.624+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:12.624+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:12.624+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:12.624+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 84, above threshold 10. [2022-09-07T23:15:13.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 88420 ms [2022-09-07T23:15:13.669+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:13.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:13.672+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:13.672+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:13.672+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:13.672+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 85, above threshold 10. [2022-09-07T23:15:14.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 89470 ms [2022-09-07T23:15:14.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:14.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:14.721+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:14.722+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:14.722+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:14.722+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 86, above threshold 10. [2022-09-07T23:15:15.486+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:15:15.486+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:15:15.486+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:15:15.491+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:15.491+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:15.491+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:15:15.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 90520 ms [2022-09-07T23:15:15.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:15.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:15.772+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:15.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:15.773+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:15.773+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 87, above threshold 10. [2022-09-07T23:15:16.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 91570 ms [2022-09-07T23:15:16.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:16.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:16.823+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:16.824+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:16.824+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:16.824+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 88, above threshold 10. [2022-09-07T23:15:17.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 92620 ms [2022-09-07T23:15:17.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:17.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:17.872+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:17.875+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:17.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:17.879+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 89, above threshold 10. [2022-09-07T23:15:18.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 93670 ms [2022-09-07T23:15:18.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:18.920+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:18.923+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:18.924+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:18.924+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:18.924+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 90, above threshold 10. [2022-09-07T23:15:19.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 94720 ms [2022-09-07T23:15:19.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:19.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:19.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:19.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:19.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:19.973+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 91, above threshold 10. [2022-09-07T23:15:21.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 95770 ms [2022-09-07T23:15:21.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:21.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:21.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:21.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:21.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:21.023+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 92, above threshold 10. [2022-09-07T23:15:22.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 96820 ms [2022-09-07T23:15:22.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:22.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:22.072+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:22.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:22.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:22.073+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 93, above threshold 10. [2022-09-07T23:15:23.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 97870 ms [2022-09-07T23:15:23.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:23.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:23.121+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:23.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:23.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:23.122+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 94, above threshold 10. [2022-09-07T23:15:24.170+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 98922 ms [2022-09-07T23:15:24.171+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:24.171+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:24.177+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:24.177+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:24.178+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:24.178+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 95, above threshold 10. [2022-09-07T23:15:25.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 99977 ms [2022-09-07T23:15:25.225+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:25.230+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:25.234+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:25.235+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:25.235+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:25.235+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 96, above threshold 10. [2022-09-07T23:15:26.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 101020 ms [2022-09-07T23:15:26.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:26.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:26.271+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:26.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:26.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:26.272+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 97, above threshold 10. [2022-09-07T23:15:27.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 102070 ms [2022-09-07T23:15:27.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:27.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:27.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:27.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:27.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:27.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 98, above threshold 10. [2022-09-07T23:15:28.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 103120 ms [2022-09-07T23:15:28.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:28.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:28.372+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:28.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:28.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:28.373+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 99, above threshold 10. [2022-09-07T23:15:29.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 104170 ms [2022-09-07T23:15:29.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:29.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:29.421+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:29.422+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:29.422+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:29.422+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 100, above threshold 10. [2022-09-07T23:15:30.469+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 105221 ms [2022-09-07T23:15:30.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:30.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:30.472+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:30.473+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:30.473+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:30.473+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 101, above threshold 10. [2022-09-07T23:15:30.492+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:15:30.493+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:15:30.494+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:15:30.497+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:30.497+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:30.498+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:15:31.519+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 106270 ms [2022-09-07T23:15:31.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:31.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:31.521+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:31.522+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:31.522+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:31.522+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 102, above threshold 10. [2022-09-07T23:15:32.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 107321 ms [2022-09-07T23:15:32.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:32.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:32.575+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:32.576+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:32.576+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:32.576+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 103, above threshold 10. [2022-09-07T23:15:33.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 108370 ms [2022-09-07T23:15:33.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:33.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:33.622+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:33.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:33.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:33.622+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 104, above threshold 10. [2022-09-07T23:15:34.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 109420 ms [2022-09-07T23:15:34.669+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:34.670+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:34.672+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:34.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:34.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:34.674+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 105, above threshold 10. [2022-09-07T23:15:35.720+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 110472 ms [2022-09-07T23:15:35.721+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:35.721+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:35.726+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:35.727+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:35.728+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:35.728+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 106, above threshold 10. [2022-09-07T23:15:36.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 111521 ms [2022-09-07T23:15:36.770+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:36.770+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:36.774+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:36.774+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:36.775+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:36.775+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 107, above threshold 10. [2022-09-07T23:15:37.821+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 112573 ms [2022-09-07T23:15:37.822+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:37.822+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:37.837+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:37.838+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:37.838+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:37.838+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 108, above threshold 10. [2022-09-07T23:15:38.873+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 113625 ms [2022-09-07T23:15:38.873+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:38.873+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:38.882+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:38.886+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:38.886+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:38.886+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 109, above threshold 10. [2022-09-07T23:15:39.921+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 114673 ms [2022-09-07T23:15:39.921+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:39.921+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:39.929+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:39.930+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:39.930+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:39.930+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 110, above threshold 10. [2022-09-07T23:15:40.970+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 115721 ms [2022-09-07T23:15:40.970+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:40.970+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:40.983+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:40.984+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:40.984+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:40.984+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 111, above threshold 10. [2022-09-07T23:15:42.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 116771 ms [2022-09-07T23:15:42.020+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:42.020+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:42.027+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:42.028+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:42.028+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:42.028+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 112, above threshold 10. [2022-09-07T23:15:43.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 117821 ms [2022-09-07T23:15:43.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:43.070+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:43.076+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:43.077+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:43.077+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:43.077+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 113, above threshold 10. [2022-09-07T23:15:44.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 118870 ms [2022-09-07T23:15:44.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:44.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:44.122+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:44.123+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:44.123+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-09-07T23:15:44.123+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 114, above threshold 10. [2022-09-07T23:15:45.166+00:00|WARN|PdpMessageHandler|Timer-0] not registered: object:engine/apex/handler [2022-09-07T23:15:45.167+00:00|INFO|network|Timer-0] [OUT|DMAAP|POLICY-PDP-PAP] {"pdpType":"apex","state":"PASSIVE","healthy":"HEALTHY","description":"Pdp Heartbeat","statistics":{"pdpInstanceId":"apex-59d63de8-902b-461a-ad49-275bc8a197f5","timeStamp":"2022-09-07T23:15:45.166421Z","pdpGroupName":"defaultGroup","policyExecutedCount":0,"policyExecutedSuccessCount":0,"policyExecutedFailCount":0,"policyDeployCount":0,"policyDeploySuccessCount":0,"policyDeployFailCount":0,"policyUndeployCount":0,"policyUndeploySuccessCount":0,"policyUndeployFailCount":0},"messageName":"PDP_STATUS","requestId":"aec1f08e-783a-48c3-afd5-70f263f198a2","timestampMs":1662592545166,"name":"apex-59d63de8-902b-461a-ad49-275bc8a197f5","pdpGroup":"defaultGroup"} [2022-09-07T23:15:45.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 119920 ms [2022-09-07T23:15:45.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:45.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:45.173+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:45.174+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:45.174+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:45.174+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 115, above threshold 10. [2022-09-07T23:15:45.498+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:15:45.498+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:15:45.499+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:15:45.505+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:45.505+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:45.505+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:15:46.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 120970 ms [2022-09-07T23:15:46.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:46.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:46.222+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:46.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:46.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:46.223+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 116, above threshold 10. [2022-09-07T23:15:47.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 122020 ms [2022-09-07T23:15:47.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:47.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:47.274+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:47.275+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:47.275+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:47.275+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 117, above threshold 10. [2022-09-07T23:15:48.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 123070 ms [2022-09-07T23:15:48.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:48.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:48.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:48.323+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:48.323+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:48.323+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 118, above threshold 10. [2022-09-07T23:15:49.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 124120 ms [2022-09-07T23:15:49.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:49.370+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:49.374+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:49.375+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:49.375+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:49.375+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 119, above threshold 10. [2022-09-07T23:15:50.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 125170 ms [2022-09-07T23:15:50.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:50.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:50.423+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:50.424+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:50.424+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:50.424+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 120, above threshold 10. [2022-09-07T23:15:51.474+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 126226 ms [2022-09-07T23:15:51.474+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:51.474+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:51.479+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:51.480+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:51.480+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:51.480+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 121, above threshold 10. [2022-09-07T23:15:52.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 127270 ms [2022-09-07T23:15:52.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:52.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:52.524+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:52.524+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:52.524+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:52.524+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 122, above threshold 10. [2022-09-07T23:15:53.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 128321 ms [2022-09-07T23:15:53.570+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:53.571+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:53.593+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:53.594+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:53.594+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:53.594+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 123, above threshold 10. [2022-09-07T23:15:54.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 129370 ms [2022-09-07T23:15:54.620+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:54.620+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:54.633+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:54.635+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:54.636+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:54.636+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 124, above threshold 10. [2022-09-07T23:15:55.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 130420 ms [2022-09-07T23:15:55.670+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:55.670+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:55.673+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:55.674+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:55.674+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:55.674+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 125, above threshold 10. [2022-09-07T23:15:56.719+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 131471 ms [2022-09-07T23:15:56.720+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:56.720+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:56.725+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:56.726+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:56.726+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:56.726+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 126, above threshold 10. [2022-09-07T23:15:57.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 132520 ms [2022-09-07T23:15:57.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:57.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:57.779+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:57.780+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:57.780+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:57.780+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 127, above threshold 10. [2022-09-07T23:15:58.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 133571 ms [2022-09-07T23:15:58.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:58.820+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:58.824+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:58.825+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:58.825+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:58.825+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 128, above threshold 10. [2022-09-07T23:15:59.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 134620 ms [2022-09-07T23:15:59.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:15:59.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:15:59.872+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:15:59.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:15:59.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:15:59.872+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 129, above threshold 10. [2022-09-07T23:16:00.512+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:16:00.513+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:16:00.513+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:16:00.525+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:00.526+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:00.527+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:16:00.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 135670 ms [2022-09-07T23:16:00.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:00.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:00.926+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:00.926+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:00.926+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:00.926+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 130, above threshold 10. [2022-09-07T23:16:01.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 136720 ms [2022-09-07T23:16:01.970+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:01.970+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:01.976+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:01.977+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:01.978+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:01.978+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 131, above threshold 10. [2022-09-07T23:16:03.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 137770 ms [2022-09-07T23:16:03.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:03.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:03.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:03.022+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:03.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:03.023+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 132, above threshold 10. [2022-09-07T23:16:04.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 138820 ms [2022-09-07T23:16:04.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:04.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:04.072+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:04.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:04.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:04.072+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 133, above threshold 10. [2022-09-07T23:16:05.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 139870 ms [2022-09-07T23:16:05.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:05.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:05.122+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:05.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:05.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:05.122+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 134, above threshold 10. [2022-09-07T23:16:06.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 140920 ms [2022-09-07T23:16:06.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:06.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:06.172+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:06.173+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:06.173+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:06.173+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 135, above threshold 10. [2022-09-07T23:16:07.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 141970 ms [2022-09-07T23:16:07.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:07.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:07.222+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:07.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:07.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:07.223+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 136, above threshold 10. [2022-09-07T23:16:08.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 143021 ms [2022-09-07T23:16:08.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:08.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:08.274+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:08.274+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:08.274+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:08.274+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 137, above threshold 10. [2022-09-07T23:16:09.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 144070 ms [2022-09-07T23:16:09.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:09.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:09.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:09.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:09.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:09.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 138, above threshold 10. [2022-09-07T23:16:10.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 145120 ms [2022-09-07T23:16:10.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:10.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:10.372+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:10.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:10.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:10.373+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 139, above threshold 10. [2022-09-07T23:16:11.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 146170 ms [2022-09-07T23:16:11.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:11.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:11.423+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:11.423+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:11.423+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:11.423+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 140, above threshold 10. [2022-09-07T23:16:12.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 147220 ms [2022-09-07T23:16:12.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:12.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:12.479+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:12.480+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:12.480+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:12.480+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 141, above threshold 10. [2022-09-07T23:16:13.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 148270 ms [2022-09-07T23:16:13.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:13.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:13.522+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:13.523+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:13.523+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:13.523+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 142, above threshold 10. [2022-09-07T23:16:14.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 149321 ms [2022-09-07T23:16:14.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:14.570+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:14.574+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:14.575+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:14.575+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:14.575+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 143, above threshold 10. [2022-09-07T23:16:15.529+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:16:15.529+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:16:15.531+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:16:15.537+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:15.537+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:15.538+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:16:15.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 150371 ms [2022-09-07T23:16:15.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:15.620+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:15.626+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:15.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:15.630+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:15.630+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 144, above threshold 10. [2022-09-07T23:16:16.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 151420 ms [2022-09-07T23:16:16.669+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:16.670+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:16.673+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:16.674+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:16.674+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:16.674+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 145, above threshold 10. [2022-09-07T23:16:17.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 152470 ms [2022-09-07T23:16:17.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:17.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:17.724+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:17.725+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:17.725+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:17.725+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 146, above threshold 10. [2022-09-07T23:16:18.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 153521 ms [2022-09-07T23:16:18.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:18.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:18.774+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:18.774+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:18.775+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:18.775+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 147, above threshold 10. [2022-09-07T23:16:19.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 154571 ms [2022-09-07T23:16:19.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:19.820+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:19.825+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:19.826+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:19.826+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:19.826+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 148, above threshold 10. [2022-09-07T23:16:20.869+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 155620 ms [2022-09-07T23:16:20.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:20.870+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:20.872+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:20.874+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:20.874+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:20.875+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 149, above threshold 10. [2022-09-07T23:16:21.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 156670 ms [2022-09-07T23:16:21.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:21.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:21.925+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:21.925+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:21.925+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:21.925+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 150, above threshold 10. [2022-09-07T23:16:22.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 157720 ms [2022-09-07T23:16:22.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:22.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:22.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:22.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:22.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:22.972+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 151, above threshold 10. [2022-09-07T23:16:24.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 158770 ms [2022-09-07T23:16:24.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:24.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:24.023+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:24.024+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:24.024+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:24.024+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 152, above threshold 10. [2022-09-07T23:16:25.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 159820 ms [2022-09-07T23:16:25.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:25.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:25.072+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:25.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:25.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:25.073+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 153, above threshold 10. [2022-09-07T23:16:26.123+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 160875 ms [2022-09-07T23:16:26.123+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:26.124+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:26.127+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:26.128+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:26.128+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:26.128+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 154, above threshold 10. [2022-09-07T23:16:27.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 161920 ms [2022-09-07T23:16:27.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:27.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:27.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:27.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:27.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:27.172+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 155, above threshold 10. [2022-09-07T23:16:28.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 162970 ms [2022-09-07T23:16:28.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:28.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:28.222+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:28.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:28.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:28.223+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 156, above threshold 10. [2022-09-07T23:16:29.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 164020 ms [2022-09-07T23:16:29.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:29.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:29.272+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:29.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:29.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:29.272+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 157, above threshold 10. [2022-09-07T23:16:30.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 165071 ms [2022-09-07T23:16:30.320+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:30.323+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:30.330+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:30.331+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:30.331+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:30.331+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 158, above threshold 10. [2022-09-07T23:16:30.538+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:16:30.539+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:16:30.539+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:16:30.542+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:30.542+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:30.542+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:16:31.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 166120 ms [2022-09-07T23:16:31.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:31.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:31.374+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:31.374+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:31.374+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:31.374+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 159, above threshold 10. [2022-09-07T23:16:32.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 167170 ms [2022-09-07T23:16:32.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:32.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:32.426+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:32.427+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:32.427+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:32.427+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 160, above threshold 10. [2022-09-07T23:16:33.473+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 168225 ms [2022-09-07T23:16:33.474+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:33.474+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:33.478+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:33.478+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:33.478+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:33.478+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 161, above threshold 10. [2022-09-07T23:16:34.521+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 169272 ms [2022-09-07T23:16:34.521+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:34.521+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:34.526+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:34.528+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:34.528+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:34.528+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 162, above threshold 10. [2022-09-07T23:16:35.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 170321 ms [2022-09-07T23:16:35.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:35.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:35.649+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:35.649+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:35.649+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:35.649+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 163, above threshold 10. [2022-09-07T23:16:36.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 171421 ms [2022-09-07T23:16:36.673+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:36.673+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:36.701+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:36.712+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:36.712+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:36.712+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 164, above threshold 10. [2022-09-07T23:16:37.721+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 172473 ms [2022-09-07T23:16:37.721+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:37.721+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:37.728+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:37.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:37.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:37.729+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 165, above threshold 10. [2022-09-07T23:16:38.770+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 173522 ms [2022-09-07T23:16:38.770+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:38.771+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:38.774+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:38.774+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:38.774+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:38.774+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 166, above threshold 10. [2022-09-07T23:16:39.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 174570 ms [2022-09-07T23:16:39.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:39.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:39.824+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:39.824+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:39.824+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:39.824+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 167, above threshold 10. [2022-09-07T23:16:40.869+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 175621 ms [2022-09-07T23:16:40.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:40.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:40.873+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:40.874+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:40.874+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:40.874+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 168, above threshold 10. [2022-09-07T23:16:41.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 176670 ms [2022-09-07T23:16:41.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:41.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:41.923+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:41.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:41.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:41.923+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 169, above threshold 10. [2022-09-07T23:16:42.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 177720 ms [2022-09-07T23:16:42.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:42.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:42.974+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:42.974+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:42.975+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:42.975+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 170, above threshold 10. [2022-09-07T23:16:44.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 178770 ms [2022-09-07T23:16:44.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:44.020+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:44.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:44.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:44.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:44.023+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 171, above threshold 10. [2022-09-07T23:16:45.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 179820 ms [2022-09-07T23:16:45.070+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:45.070+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:45.073+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:45.074+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:45.074+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:45.074+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 172, above threshold 10. [2022-09-07T23:16:45.543+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:16:45.543+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:16:45.543+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:16:45.548+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:45.548+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:45.548+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:16:46.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 180870 ms [2022-09-07T23:16:46.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:46.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:46.124+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:46.124+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:46.124+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:46.124+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 173, above threshold 10. [2022-09-07T23:16:47.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 181920 ms [2022-09-07T23:16:47.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:47.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:47.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:47.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:47.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:47.172+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 174, above threshold 10. [2022-09-07T23:16:48.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 182970 ms [2022-09-07T23:16:48.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:48.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:48.226+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:48.226+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:48.227+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:48.227+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 175, above threshold 10. [2022-09-07T23:16:49.271+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 184023 ms [2022-09-07T23:16:49.271+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:49.271+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:49.278+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:49.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:49.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:49.279+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 176, above threshold 10. [2022-09-07T23:16:50.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 185070 ms [2022-09-07T23:16:50.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:50.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:50.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:50.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:50.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:50.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 177, above threshold 10. [2022-09-07T23:16:51.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 186120 ms [2022-09-07T23:16:51.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:51.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:51.372+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:51.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:51.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:51.373+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 178, above threshold 10. [2022-09-07T23:16:52.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 187170 ms [2022-09-07T23:16:52.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:52.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:52.423+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:52.424+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:52.424+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:52.424+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 179, above threshold 10. [2022-09-07T23:16:53.469+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 188220 ms [2022-09-07T23:16:53.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:53.470+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:53.473+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:53.473+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:53.474+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:53.474+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 180, above threshold 10. [2022-09-07T23:16:54.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 189270 ms [2022-09-07T23:16:54.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:54.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:54.521+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:54.522+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:54.522+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:54.522+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 181, above threshold 10. [2022-09-07T23:16:55.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 190320 ms [2022-09-07T23:16:55.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:55.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:55.572+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:55.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:55.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:55.572+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 182, above threshold 10. [2022-09-07T23:16:56.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 191370 ms [2022-09-07T23:16:56.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:56.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:56.621+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:56.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:56.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:56.622+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 183, above threshold 10. [2022-09-07T23:16:57.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 192420 ms [2022-09-07T23:16:57.669+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:57.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:57.672+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:57.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:57.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:57.673+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 184, above threshold 10. [2022-09-07T23:16:58.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 193470 ms [2022-09-07T23:16:58.723+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:58.723+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:58.726+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:58.727+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:58.727+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:58.727+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 185, above threshold 10. [2022-09-07T23:16:59.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 194520 ms [2022-09-07T23:16:59.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:16:59.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:16:59.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:16:59.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:16:59.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:16:59.772+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 186, above threshold 10. [2022-09-07T23:17:00.548+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:17:00.549+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:17:00.549+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:17:00.552+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:00.553+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:00.553+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:17:00.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 195570 ms [2022-09-07T23:17:00.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:00.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:00.825+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:00.826+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:00.826+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:00.826+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 187, above threshold 10. [2022-09-07T23:17:01.869+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 196620 ms [2022-09-07T23:17:01.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:01.870+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:01.873+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:01.874+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:01.874+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:01.874+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 188, above threshold 10. [2022-09-07T23:17:02.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 197670 ms [2022-09-07T23:17:02.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:02.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:02.922+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:02.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:02.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:02.922+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 189, above threshold 10. [2022-09-07T23:17:03.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 198721 ms [2022-09-07T23:17:03.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:03.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:03.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:03.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:03.974+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:03.974+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 190, above threshold 10. [2022-09-07T23:17:04.977+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 199729 ms [2022-09-07T23:17:04.981+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:04.981+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:04.985+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:04.986+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:04.986+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:04.986+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 191, above threshold 10. [2022-09-07T23:17:06.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 200770 ms [2022-09-07T23:17:06.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:06.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:06.023+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:06.024+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:06.024+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:06.024+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 192, above threshold 10. [2022-09-07T23:17:07.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 201820 ms [2022-09-07T23:17:07.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:07.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:07.072+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:07.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:07.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:07.073+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 193, above threshold 10. [2022-09-07T23:17:08.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 202870 ms [2022-09-07T23:17:08.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:08.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:08.126+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:08.127+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:08.127+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:08.127+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 194, above threshold 10. [2022-09-07T23:17:09.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 203920 ms [2022-09-07T23:17:09.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:09.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:09.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:09.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:09.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:09.172+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 195, above threshold 10. [2022-09-07T23:17:10.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 204970 ms [2022-09-07T23:17:10.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:10.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:10.222+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:10.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:10.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:10.223+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 196, above threshold 10. [2022-09-07T23:17:11.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 206020 ms [2022-09-07T23:17:11.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:11.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:11.272+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:11.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:11.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:11.273+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 197, above threshold 10. [2022-09-07T23:17:12.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 207070 ms [2022-09-07T23:17:12.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:12.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:12.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:12.323+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:12.323+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:12.323+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 198, above threshold 10. [2022-09-07T23:17:13.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 208120 ms [2022-09-07T23:17:13.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:13.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:13.373+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:13.374+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:13.374+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:13.374+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 199, above threshold 10. [2022-09-07T23:17:14.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 209171 ms [2022-09-07T23:17:14.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:14.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:14.423+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:14.423+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:14.423+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:14.423+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 200, above threshold 10. [2022-09-07T23:17:15.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 210220 ms [2022-09-07T23:17:15.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:15.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:15.471+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:15.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:15.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:15.472+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 201, above threshold 10. [2022-09-07T23:17:15.553+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:17:15.553+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:17:15.554+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:17:15.557+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:15.558+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:15.558+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:17:16.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 211270 ms [2022-09-07T23:17:16.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:16.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:16.523+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:16.523+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:16.524+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:16.524+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 202, above threshold 10. [2022-09-07T23:17:17.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 212320 ms [2022-09-07T23:17:17.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:17.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:17.573+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:17.574+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:17.575+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:17.575+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 203, above threshold 10. [2022-09-07T23:17:18.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 213370 ms [2022-09-07T23:17:18.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:18.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:18.623+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:18.624+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:18.624+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:18.624+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 204, above threshold 10. [2022-09-07T23:17:19.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 214420 ms [2022-09-07T23:17:19.669+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:19.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:19.674+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:19.675+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:19.675+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:19.675+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 205, above threshold 10. [2022-09-07T23:17:20.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 215470 ms [2022-09-07T23:17:20.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:20.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:20.721+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:20.721+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:20.722+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:20.722+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 206, above threshold 10. [2022-09-07T23:17:21.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 216520 ms [2022-09-07T23:17:21.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:21.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:21.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:21.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:21.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:21.772+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 207, above threshold 10. [2022-09-07T23:17:22.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 217571 ms [2022-09-07T23:17:22.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:22.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:22.822+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:22.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:22.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:22.822+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 208, above threshold 10. [2022-09-07T23:17:23.869+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 218621 ms [2022-09-07T23:17:23.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:23.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:23.872+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:23.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:23.873+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:23.873+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 209, above threshold 10. [2022-09-07T23:17:24.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 219671 ms [2022-09-07T23:17:24.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:24.920+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:24.928+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:24.945+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:24.945+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:24.945+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 210, above threshold 10. [2022-09-07T23:17:25.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 220720 ms [2022-09-07T23:17:25.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:25.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:25.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:25.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:25.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:25.973+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 211, above threshold 10. [2022-09-07T23:17:27.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 221770 ms [2022-09-07T23:17:27.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:27.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:27.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:27.024+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:27.024+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:27.024+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 212, above threshold 10. [2022-09-07T23:17:28.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 222820 ms [2022-09-07T23:17:28.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:28.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:28.073+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:28.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:28.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:28.073+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 213, above threshold 10. [2022-09-07T23:17:29.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 223870 ms [2022-09-07T23:17:29.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:29.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:29.121+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:29.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:29.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:29.121+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 214, above threshold 10. [2022-09-07T23:17:30.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 224920 ms [2022-09-07T23:17:30.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:30.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:30.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:30.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:30.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:30.172+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 215, above threshold 10. [2022-09-07T23:17:30.558+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=ea90a92d-6c84-4b83-b988-d31fd03120a0, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:201) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-09-07T23:17:30.559+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 [2022-09-07T23:17:30.559+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/ea90a92d-6c84-4b83-b988-d31fd03120a0/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-09-07T23:17:30.563+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:30.563+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:30.564+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-09-07T23:17:31.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 225970 ms [2022-09-07T23:17:31.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:31.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:31.222+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:31.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:31.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:31.222+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 216, above threshold 10. [2022-09-07T23:17:32.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 227021 ms [2022-09-07T23:17:32.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:32.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:32.273+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:32.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:32.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:32.273+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 217, above threshold 10. [2022-09-07T23:17:33.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 228070 ms [2022-09-07T23:17:33.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:33.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:33.321+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:33.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:33.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:33.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 218, above threshold 10. [2022-09-07T23:17:34.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 229120 ms [2022-09-07T23:17:34.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:34.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:34.378+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:34.378+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:34.378+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:34.378+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 219, above threshold 10. [2022-09-07T23:17:35.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 230171 ms [2022-09-07T23:17:35.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-09-07T23:17:35.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-09-07T23:17:35.424+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-09-07T23:17:35.424+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-09-07T23:17:35.424+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-09-07T23:17:35.424+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 220, above threshold 10. [2022-09-