overriding logback.xml apexApps.sh: running application 'onappf' with command 'java -Dlogback.configurationFile=/opt/app/policy/apex-pdp/etc/logback.xml -cp /opt/app/policy/apex-pdp/etc:/opt/app/policy/apex-pdp/etc/hazelcast:/opt/app/policy/apex-pdp/etc/infinispan:/opt/app/policy/apex-pdp/lib/* -Djavax.net.ssl.keyStore=/opt/app/policy/apex-pdp/etc/ssl/policy-keystore -Djavax.net.ssl.keyStorePassword=Pol1cy_0nap -Djavax.net.ssl.trustStore=/opt/app/policy/apex-pdp/etc/ssl/policy-truststore -Djavax.net.ssl.trustStorePassword=Pol1cy_0nap -Dlogback.configurationFile=/opt/app/policy/apex-pdp/etc/logback.xml -Dhazelcast.config=/opt/app/policy/apex-pdp/etc/hazelcast.xml -Dhazelcast.mancenter.enabled=false org.onap.policy.apex.services.onappf.ApexStarterMain -c /home/apexuser/config/OnapPfConfig.json' [2022-11-22T22:53:49.579+00:00|INFO|ApexStarterMain|main] In ApexStarter with parameters [-c, /home/apexuser/config/OnapPfConfig.json] [2022-11-22T22:53:50.573+00:00|INFO|SingleThreadedDmaapTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: INITTED [2022-11-22T22:53:50.747+00:00|INFO|ServiceManager|main] service manager starting [2022-11-22T22:53:50.748+00:00|INFO|ServiceManager|main] service manager starting topics [2022-11-22T22:53:50.763+00:00|INFO|SingleThreadedBusTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: starting [2022-11-22T22:53:50.765+00:00|INFO|SingleThreadedDmaapTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: INITTED [2022-11-22T22:53:50.772+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:53:50.781+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:53:50.767+00:00|INFO|InlineBusTopicSink|main] InlineDmaapTopicSink [userName=null, password=null, getTopicCommInfrastructure()=DMAAP, toString()=InlineBusTopicSink [partitionId=8b174273-2cb7-4010-87a9-c9e54691d86a, alive=false, publisher=null]]: starting [2022-11-22T22:53:50.800+00:00|INFO|InlineDmaapTopicSink|main] InlineDmaapTopicSink [userName=null, password=null, getTopicCommInfrastructure()=DMAAP, toString()=InlineBusTopicSink [partitionId=8b174273-2cb7-4010-87a9-c9e54691d86a, alive=false, publisher=CambriaPublisherWrapper []]]: DMAAP SINK created [2022-11-22T22:53:50.801+00:00|INFO|ServiceManager|main] service manager starting set alive [2022-11-22T22:53:50.801+00:00|INFO|ServiceManager|main] service manager starting register pdp status context object [2022-11-22T22:53:50.805+00:00|INFO|ServiceManager|main] service manager starting topic sinks [2022-11-22T22:53:50.806+00:00|INFO|ServiceManager|main] service manager starting Pdp Status publisher [2022-11-22T22:53:50.921+00:00|INFO|ServiceManager|main] service manager starting Register pdp update listener [2022-11-22T22:53:50.922+00:00|INFO|ServiceManager|main] service manager starting Register pdp state change request dispatcher [2022-11-22T22:53:50.922+00:00|INFO|ServiceManager|main] service manager starting Message Dispatcher [2022-11-22T22:53:50.923+00:00|INFO|TopicBase|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: registering org.onap.policy.common.endpoints.listeners.MessageTypeDispatcher@305f031 [2022-11-22T22:53:50.923+00:00|WARN|PdpMessageHandler|Timer-0] not registered: object:engine/apex/handler [2022-11-22T22:53:50.924+00:00|INFO|SingleThreadedBusTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: register: start not attempted [2022-11-22T22:53:50.925+00:00|INFO|ServiceManager|main] service manager starting Create REST server [2022-11-22T22:53:50.955+00:00|INFO|OrderedServiceImpl|Timer-0] ***** OrderedServiceImpl implementers: [] [2022-11-22T22:53:50.958+00:00|INFO|network|Timer-0] [OUT|DMAAP|POLICY-PDP-PAP] {"pdpType":"apex","state":"PASSIVE","healthy":"HEALTHY","description":"Pdp Heartbeat","statistics":{"pdpInstanceId":"apex-082c82bb-bec8-4e9d-9651-e46b70119585","timeStamp":"2022-11-22T22:53:50.926896Z","pdpGroupName":"defaultGroup","policyExecutedCount":0,"policyExecutedSuccessCount":0,"policyExecutedFailCount":0,"policyDeployCount":0,"policyDeploySuccessCount":0,"policyDeployFailCount":0,"policyUndeployCount":0,"policyUndeploySuccessCount":0,"policyUndeployFailCount":0},"messageName":"PDP_STATUS","requestId":"84d55249-8fbb-46ba-b0fd-f8a7137f2cc6","timestampMs":1669157630922,"name":"apex-082c82bb-bec8-4e9d-9651-e46b70119585","pdpGroup":"defaultGroup"} [2022-11-22T22:53:51.000+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 39 ms [2022-11-22T22:53:51.003+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:53:51.022+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:51.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:51.025+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:51.026+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:53:51.029+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:53:51.030+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:53:51.058+00:00|INFO|log|main] Logging initialized @2647ms to org.eclipse.jetty.util.log.Slf4jLog [2022-11-22T22:53:51.270+00:00|INFO|ServiceManager|main] service manager starting Rest Server [2022-11-22T22:53:51.271+00:00|INFO|ServiceManager|main] service manager starting [2022-11-22T22:53:51.271+00:00|INFO|ServiceManager|main] service manager starting REST RestServerParameters [2022-11-22T22:53:51.271+00:00|INFO|JettyServletServer|main] JettyJerseyServer [Jerseyservlets={/metrics=io.prometheus.client.exporter.MetricsServlet-51abf713==io.prometheus.client.exporter.MetricsServlet@aabd17f5{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-63a5e46c==org.glassfish.jersey.servlet.ServletContainer@f4f7771b{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-3caa4757==io.swagger.jersey.config.JerseyJaxrsConfig@c9481d48{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}}, swaggerId=swagger-6969, toString()=JettyServletServer(name=RestServerParameters, host=0.0.0.0, port=6969, user=healthcheck, password=zb!XztG34, contextPath=/, jettyServer=Server@676f0a60{STOPPED}[9.4.48.v20220622], context=o.e.j.s.ServletContextHandler@5d10455d{/,null,STOPPED}, connector=RestServerParameters@5ed190be{HTTP/1.1, (http/1.1)}{0.0.0.0:6969}, jettyThread=null, servlets={/metrics=io.prometheus.client.exporter.MetricsServlet-51abf713==io.prometheus.client.exporter.MetricsServlet@aabd17f5{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-63a5e46c==org.glassfish.jersey.servlet.ServletContainer@f4f7771b{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-3caa4757==io.swagger.jersey.config.JerseyJaxrsConfig@c9481d48{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}})]: STARTING [2022-11-22T22:53:51.321+00:00|INFO|ServiceManager|main] service manager started [2022-11-22T22:53:51.321+00:00|INFO|ServiceManager|main] service manager started [2022-11-22T22:53:51.322+00:00|INFO|ApexStarterMain|main] Started policy-apex-pdp service successfully. [2022-11-22T22:53:51.321+00:00|INFO|JettyServletServer|RestServerParameters-6969] JettyJerseyServer [Jerseyservlets={/metrics=io.prometheus.client.exporter.MetricsServlet-51abf713==io.prometheus.client.exporter.MetricsServlet@aabd17f5{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-63a5e46c==org.glassfish.jersey.servlet.ServletContainer@f4f7771b{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-3caa4757==io.swagger.jersey.config.JerseyJaxrsConfig@c9481d48{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}}, swaggerId=swagger-6969, toString()=JettyServletServer(name=RestServerParameters, host=0.0.0.0, port=6969, user=healthcheck, password=zb!XztG34, contextPath=/, jettyServer=Server@676f0a60{STOPPED}[9.4.48.v20220622], context=o.e.j.s.ServletContextHandler@5d10455d{/,null,STOPPED}, connector=RestServerParameters@5ed190be{HTTP/1.1, (http/1.1)}{0.0.0.0:6969}, jettyThread=Thread[RestServerParameters-6969,5,main], servlets={/metrics=io.prometheus.client.exporter.MetricsServlet-51abf713==io.prometheus.client.exporter.MetricsServlet@aabd17f5{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-63a5e46c==org.glassfish.jersey.servlet.ServletContainer@f4f7771b{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-3caa4757==io.swagger.jersey.config.JerseyJaxrsConfig@c9481d48{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}})]: STARTING [2022-11-22T22:53:51.333+00:00|INFO|Server|RestServerParameters-6969] jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.15+10-alpine-r0 [2022-11-22T22:53:51.385+00:00|INFO|session|RestServerParameters-6969] DefaultSessionIdManager workerName=node0 [2022-11-22T22:53:51.385+00:00|INFO|session|RestServerParameters-6969] No SessionScavenger set, using defaults [2022-11-22T22:53:51.387+00:00|INFO|session|RestServerParameters-6969] node0 Scavenging every 660000ms [2022-11-22T22:53:52.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 1089 ms [2022-11-22T22:53:52.051+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:53:52.051+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:53:52.054+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:52.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:53:52.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.sun.xml.bind.v2.runtime.reflect.opt.Injector (file:/opt/app/policy/apex-pdp/lib/jaxb-impl-2.3.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) WARNING: Please consider reporting this to the maintainers of com.sun.xml.bind.v2.runtime.reflect.opt.Injector WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [2022-11-22T22:53:52.466+00:00|INFO|YamlMessageBodyHandler|RestServerParameters-6969] Accepting YAML for REST calls [2022-11-22T22:53:52.466+00:00|INFO|GsonMessageBodyHandler|RestServerParameters-6969] Using GSON for REST calls [2022-11-22T22:53:52.682+00:00|INFO|ContextHandler|RestServerParameters-6969] Started o.e.j.s.ServletContextHandler@5d10455d{/,null,AVAILABLE} [2022-11-22T22:53:52.700+00:00|INFO|AbstractConnector|RestServerParameters-6969] Started RestServerParameters@5ed190be{HTTP/1.1, (http/1.1)}{0.0.0.0:6969} [2022-11-22T22:53:52.701+00:00|INFO|Server|RestServerParameters-6969] Started @4290ms [2022-11-22T22:53:53.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 2139 ms [2022-11-22T22:53:53.100+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:53:53.101+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:53:53.103+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:53.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:53:53.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:53:54.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 3189 ms [2022-11-22T22:53:54.150+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:53:54.151+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:53:54.153+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:54.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:53:54.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:53:55.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 4239 ms [2022-11-22T22:53:55.200+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:53:55.201+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:53:55.203+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:55.203+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:53:55.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:53:56.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 5289 ms [2022-11-22T22:53:56.251+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:53:56.251+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:53:56.253+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:56.254+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:53:56.254+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:53:57.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 6339 ms [2022-11-22T22:53:57.301+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:53:57.301+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:53:57.304+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:57.304+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:53:57.304+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:53:58.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 7389 ms [2022-11-22T22:53:58.351+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:53:58.351+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:53:58.353+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:58.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:53:58.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:53:59.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 8439 ms [2022-11-22T22:53:59.400+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:53:59.401+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:53:59.403+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:53:59.404+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:53:59.404+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:00.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 9489 ms [2022-11-22T22:54:00.450+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:00.451+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:00.453+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:00.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:00.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:01.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 10539 ms [2022-11-22T22:54:01.501+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:01.501+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:01.503+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:01.504+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:01.504+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:01.504+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 11, above threshold 10. [2022-11-22T22:54:02.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 11589 ms [2022-11-22T22:54:02.550+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:02.551+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:02.553+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:02.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:02.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:02.554+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 12, above threshold 10. [2022-11-22T22:54:03.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 12639 ms [2022-11-22T22:54:03.600+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:03.600+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:03.602+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:03.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:03.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:03.603+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 13, above threshold 10. [2022-11-22T22:54:04.650+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 13689 ms [2022-11-22T22:54:04.650+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:04.651+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:04.653+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:04.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:04.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:04.653+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 14, above threshold 10. [2022-11-22T22:54:05.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 14739 ms [2022-11-22T22:54:05.700+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:05.701+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:05.703+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:05.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:05.704+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:05.704+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 15, above threshold 10. [2022-11-22T22:54:06.027+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:54:06.028+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:54:06.028+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:54:06.031+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:06.031+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:06.031+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:54:06.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 15789 ms [2022-11-22T22:54:06.750+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:06.751+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:06.753+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:06.754+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:06.754+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:06.754+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 16, above threshold 10. [2022-11-22T22:54:07.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 16839 ms [2022-11-22T22:54:07.800+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:07.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:07.803+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:07.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:07.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:07.804+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 17, above threshold 10. [2022-11-22T22:54:08.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 17889 ms [2022-11-22T22:54:08.850+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:08.851+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:08.853+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:08.853+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:08.853+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:08.853+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 18, above threshold 10. [2022-11-22T22:54:09.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 18939 ms [2022-11-22T22:54:09.900+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:09.901+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:09.903+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:09.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:09.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:09.904+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 19, above threshold 10. [2022-11-22T22:54:10.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 19989 ms [2022-11-22T22:54:10.950+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:10.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:10.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:10.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:10.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:10.954+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 20, above threshold 10. [2022-11-22T22:54:12.000+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 21039 ms [2022-11-22T22:54:12.000+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:12.001+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:12.003+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:12.003+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:12.003+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:12.003+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 21, above threshold 10. [2022-11-22T22:54:13.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 22089 ms [2022-11-22T22:54:13.050+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:13.051+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:13.053+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:13.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:13.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:13.054+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 22, above threshold 10. [2022-11-22T22:54:14.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 23139 ms [2022-11-22T22:54:14.101+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:14.101+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:14.104+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:14.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:14.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:14.104+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23, above threshold 10. [2022-11-22T22:54:15.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 24189 ms [2022-11-22T22:54:15.150+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:15.151+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:15.153+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:15.154+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:15.154+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:15.154+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 24, above threshold 10. [2022-11-22T22:54:16.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 25239 ms [2022-11-22T22:54:16.201+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:16.201+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:16.203+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:16.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:16.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:16.204+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 25, above threshold 10. [2022-11-22T22:54:17.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 26289 ms [2022-11-22T22:54:17.251+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:17.251+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:17.254+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:17.254+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:17.254+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:17.254+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 26, above threshold 10. [2022-11-22T22:54:18.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 27339 ms [2022-11-22T22:54:18.300+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:18.300+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:18.303+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:18.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:18.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:18.303+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 27, above threshold 10. [2022-11-22T22:54:19.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 28389 ms [2022-11-22T22:54:19.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:19.351+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:19.353+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:19.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:19.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:19.354+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 28, above threshold 10. [2022-11-22T22:54:20.401+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 29440 ms [2022-11-22T22:54:20.402+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:20.402+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:20.404+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:20.405+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:20.405+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:20.405+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 29, above threshold 10. [2022-11-22T22:54:21.032+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:54:21.033+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:54:21.033+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:54:21.036+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:21.037+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:21.037+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:54:21.452+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 30490 ms [2022-11-22T22:54:21.452+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:21.453+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:21.460+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:21.461+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:21.461+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:21.462+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 30, above threshold 10. [2022-11-22T22:54:22.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 31539 ms [2022-11-22T22:54:22.501+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:22.501+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:22.503+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:22.504+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:22.504+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:22.504+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 31, above threshold 10. [2022-11-22T22:54:23.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 32589 ms [2022-11-22T22:54:23.551+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:23.551+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:23.553+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:23.553+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:23.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:23.554+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 32, above threshold 10. [2022-11-22T22:54:24.604+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 33643 ms [2022-11-22T22:54:24.604+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:24.604+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:24.612+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:24.614+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:24.614+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:24.614+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 33, above threshold 10. [2022-11-22T22:54:25.653+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 34692 ms [2022-11-22T22:54:25.654+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:25.654+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:25.656+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:25.657+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:25.657+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:25.657+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 34, above threshold 10. [2022-11-22T22:54:26.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 35739 ms [2022-11-22T22:54:26.700+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:26.701+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:26.703+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:26.704+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:26.704+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:26.704+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 35, above threshold 10. [2022-11-22T22:54:27.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 36789 ms [2022-11-22T22:54:27.751+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:27.751+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:27.754+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:27.755+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:27.755+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:27.755+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 36, above threshold 10. [2022-11-22T22:54:28.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 37839 ms [2022-11-22T22:54:28.801+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:28.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:28.805+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:28.806+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:28.806+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:28.806+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 37, above threshold 10. [2022-11-22T22:54:29.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 38889 ms [2022-11-22T22:54:29.850+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:29.850+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:29.854+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:29.854+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:29.854+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:29.854+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 38, above threshold 10. [2022-11-22T22:54:30.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 39939 ms [2022-11-22T22:54:30.900+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:30.901+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:30.904+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:30.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:30.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:30.904+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 39, above threshold 10. [2022-11-22T22:54:31.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 40989 ms [2022-11-22T22:54:31.950+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:31.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:31.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:31.955+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:31.955+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:31.955+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 40, above threshold 10. [2022-11-22T22:54:33.004+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 42043 ms [2022-11-22T22:54:33.004+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:33.004+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:33.009+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:33.009+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:33.009+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:33.009+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 41, above threshold 10. [2022-11-22T22:54:34.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 43089 ms [2022-11-22T22:54:34.051+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:34.051+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:34.054+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:34.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:34.055+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:34.055+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 42, above threshold 10. [2022-11-22T22:54:35.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 44139 ms [2022-11-22T22:54:35.100+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:35.101+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:35.103+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:35.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:35.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:35.104+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 43, above threshold 10. [2022-11-22T22:54:36.037+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:54:36.038+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:54:36.038+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:54:36.041+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:36.041+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:36.041+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:54:36.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 45189 ms [2022-11-22T22:54:36.150+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:36.150+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:36.152+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:36.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:36.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:36.153+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 44, above threshold 10. [2022-11-22T22:54:37.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 46239 ms [2022-11-22T22:54:37.200+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:37.200+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:37.202+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:37.203+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:37.203+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:37.203+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 45, above threshold 10. [2022-11-22T22:54:38.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 47289 ms [2022-11-22T22:54:38.251+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:38.251+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:38.254+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:38.254+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:38.254+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:38.254+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 46, above threshold 10. [2022-11-22T22:54:39.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 48339 ms [2022-11-22T22:54:39.300+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:39.301+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:39.303+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:39.304+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:39.304+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:39.304+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 47, above threshold 10. [2022-11-22T22:54:40.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 49389 ms [2022-11-22T22:54:40.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:40.351+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:40.353+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:40.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:40.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:40.354+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 48, above threshold 10. [2022-11-22T22:54:41.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 50439 ms [2022-11-22T22:54:41.401+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:41.402+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:41.404+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:41.404+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:41.405+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:41.405+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 49, above threshold 10. [2022-11-22T22:54:42.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 51489 ms [2022-11-22T22:54:42.451+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:42.451+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:42.453+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:42.454+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:42.454+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:42.454+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 50, above threshold 10. [2022-11-22T22:54:43.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 52539 ms [2022-11-22T22:54:43.500+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:43.501+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:43.503+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:43.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:43.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:43.503+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 51, above threshold 10. [2022-11-22T22:54:44.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 53589 ms [2022-11-22T22:54:44.550+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:44.550+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:44.553+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:44.553+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:44.553+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:44.553+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 52, above threshold 10. [2022-11-22T22:54:45.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 54639 ms [2022-11-22T22:54:45.601+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:45.601+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:45.605+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:45.606+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:45.606+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:45.606+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 53, above threshold 10. [2022-11-22T22:54:46.651+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 55689 ms [2022-11-22T22:54:46.652+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:46.652+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:46.654+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:46.655+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:46.655+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:46.655+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 54, above threshold 10. [2022-11-22T22:54:47.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 56739 ms [2022-11-22T22:54:47.700+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:47.700+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:47.704+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:47.705+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:47.705+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:47.705+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 55, above threshold 10. [2022-11-22T22:54:48.751+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 57790 ms [2022-11-22T22:54:48.751+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:48.751+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:48.756+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:48.760+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:48.760+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:48.760+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 56, above threshold 10. [2022-11-22T22:54:49.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 58839 ms [2022-11-22T22:54:49.800+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:49.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:49.803+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:49.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:49.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:49.804+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 57, above threshold 10. [2022-11-22T22:54:50.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 59889 ms [2022-11-22T22:54:50.850+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:50.851+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:50.854+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:50.854+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:50.854+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:50.855+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 58, above threshold 10. [2022-11-22T22:54:51.042+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:54:51.042+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:54:51.042+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:54:51.045+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:51.045+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:51.045+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:54:51.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 60939 ms [2022-11-22T22:54:51.900+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:51.901+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:51.910+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:51.911+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:51.911+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:51.911+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 59, above threshold 10. [2022-11-22T22:54:52.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 61989 ms [2022-11-22T22:54:52.951+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:52.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:52.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:52.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:52.955+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:52.955+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 60, above threshold 10. [2022-11-22T22:54:54.000+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 63039 ms [2022-11-22T22:54:54.001+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:54.001+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:54.004+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:54.004+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:54.004+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:54.005+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 61, above threshold 10. [2022-11-22T22:54:55.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 64089 ms [2022-11-22T22:54:55.051+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:55.051+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:55.054+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:55.055+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:55.055+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:55.055+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 62, above threshold 10. [2022-11-22T22:54:56.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 65139 ms [2022-11-22T22:54:56.101+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:56.101+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:56.105+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:56.105+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:56.106+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:56.106+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 63, above threshold 10. [2022-11-22T22:54:57.151+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 66190 ms [2022-11-22T22:54:57.151+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:57.151+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:57.158+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:57.159+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:57.159+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:57.159+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 64, above threshold 10. [2022-11-22T22:54:58.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 67239 ms [2022-11-22T22:54:58.201+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:58.201+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:58.204+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:58.205+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:58.205+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:58.205+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 65, above threshold 10. [2022-11-22T22:54:59.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 68289 ms [2022-11-22T22:54:59.251+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:54:59.251+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:54:59.254+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:54:59.255+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:54:59.255+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:54:59.255+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 66, above threshold 10. [2022-11-22T22:55:00.255+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 69294 ms [2022-11-22T22:55:00.255+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:00.255+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:00.259+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:00.259+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:00.259+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:00.259+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 67, above threshold 10. [2022-11-22T22:55:01.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 70339 ms [2022-11-22T22:55:01.300+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:01.301+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:01.303+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:01.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:01.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:01.303+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 68, above threshold 10. [2022-11-22T22:55:02.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 71389 ms [2022-11-22T22:55:02.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:02.351+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:02.355+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:02.355+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:02.355+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:02.355+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 69, above threshold 10. [2022-11-22T22:55:03.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 72439 ms [2022-11-22T22:55:03.400+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:03.401+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:03.403+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:03.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:03.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:03.404+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 70, above threshold 10. [2022-11-22T22:55:04.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 73489 ms [2022-11-22T22:55:04.450+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:04.450+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:04.452+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:04.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:04.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:04.453+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 71, above threshold 10. [2022-11-22T22:55:05.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 74539 ms [2022-11-22T22:55:05.500+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:05.500+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:05.503+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:05.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:05.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:05.503+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 72, above threshold 10. [2022-11-22T22:55:06.046+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:55:06.046+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:55:06.046+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:55:06.050+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:06.050+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:06.050+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:55:06.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 75589 ms [2022-11-22T22:55:06.551+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:06.551+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:06.554+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:06.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:06.555+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:06.555+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 73, above threshold 10. [2022-11-22T22:55:07.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 76639 ms [2022-11-22T22:55:07.600+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:07.600+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:07.603+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:07.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:07.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:07.603+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 74, above threshold 10. [2022-11-22T22:55:08.650+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 77689 ms [2022-11-22T22:55:08.650+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:08.651+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:08.653+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:08.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:08.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:08.653+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 75, above threshold 10. [2022-11-22T22:55:09.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 78739 ms [2022-11-22T22:55:09.700+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:09.701+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:09.703+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:09.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:09.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:09.703+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 76, above threshold 10. [2022-11-22T22:55:10.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 79789 ms [2022-11-22T22:55:10.750+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:10.751+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:10.753+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:10.754+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:10.754+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:10.754+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 77, above threshold 10. [2022-11-22T22:55:11.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 80839 ms [2022-11-22T22:55:11.800+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:11.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:11.803+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:11.803+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:11.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:11.804+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 78, above threshold 10. [2022-11-22T22:55:12.853+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 81892 ms [2022-11-22T22:55:12.853+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:12.853+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:12.879+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:12.880+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:12.880+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:12.880+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 79, above threshold 10. [2022-11-22T22:55:13.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 82939 ms [2022-11-22T22:55:13.900+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:13.900+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:13.903+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:13.903+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:13.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:13.904+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 80, above threshold 10. [2022-11-22T22:55:14.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 83989 ms [2022-11-22T22:55:14.950+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:14.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:14.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:14.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:14.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:14.954+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 81, above threshold 10. [2022-11-22T22:55:16.000+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 85039 ms [2022-11-22T22:55:16.000+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:16.001+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:16.004+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:16.005+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:16.005+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:16.005+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 82, above threshold 10. [2022-11-22T22:55:17.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 86089 ms [2022-11-22T22:55:17.051+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:17.051+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:17.054+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:17.055+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:17.055+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:17.055+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 83, above threshold 10. [2022-11-22T22:55:18.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 87139 ms [2022-11-22T22:55:18.101+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:18.101+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:18.103+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:18.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:18.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:18.104+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 84, above threshold 10. [2022-11-22T22:55:19.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 88189 ms [2022-11-22T22:55:19.150+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:19.151+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:19.153+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:19.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:19.154+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:19.154+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 85, above threshold 10. [2022-11-22T22:55:20.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 89239 ms [2022-11-22T22:55:20.200+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:20.201+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:20.203+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:20.203+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:20.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:20.204+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 86, above threshold 10. [2022-11-22T22:55:21.050+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:55:21.050+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:55:21.051+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:55:21.053+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:21.053+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:21.053+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:55:21.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 90289 ms [2022-11-22T22:55:21.250+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:21.250+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:21.253+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:21.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:21.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:21.253+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 87, above threshold 10. [2022-11-22T22:55:22.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 91339 ms [2022-11-22T22:55:22.300+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:22.301+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:22.306+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:22.307+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:22.307+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:22.307+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 88, above threshold 10. [2022-11-22T22:55:23.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 92389 ms [2022-11-22T22:55:23.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:23.350+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:23.353+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:23.353+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:23.353+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:23.353+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 89, above threshold 10. [2022-11-22T22:55:24.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 93439 ms [2022-11-22T22:55:24.400+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:24.401+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:24.403+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:24.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:24.404+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:24.404+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 90, above threshold 10. [2022-11-22T22:55:25.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 94489 ms [2022-11-22T22:55:25.450+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:25.451+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:25.453+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:25.454+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:25.454+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:25.454+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 91, above threshold 10. [2022-11-22T22:55:26.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 95539 ms [2022-11-22T22:55:26.500+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:26.501+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:26.504+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:26.504+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:26.504+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:26.504+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 92, above threshold 10. [2022-11-22T22:55:27.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 96589 ms [2022-11-22T22:55:27.550+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:27.551+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:27.553+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:27.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:27.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:27.554+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 93, above threshold 10. [2022-11-22T22:55:28.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 97639 ms [2022-11-22T22:55:28.600+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:28.600+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:28.603+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:28.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:28.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:28.604+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 94, above threshold 10. [2022-11-22T22:55:29.650+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 98689 ms [2022-11-22T22:55:29.650+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:29.651+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:29.653+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:29.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:29.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:29.653+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 95, above threshold 10. [2022-11-22T22:55:30.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 99739 ms [2022-11-22T22:55:30.700+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:30.700+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:30.703+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:30.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:30.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:30.703+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 96, above threshold 10. [2022-11-22T22:55:31.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 100789 ms [2022-11-22T22:55:31.750+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:31.750+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:31.753+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:31.753+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:31.753+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:31.753+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 97, above threshold 10. [2022-11-22T22:55:32.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 101839 ms [2022-11-22T22:55:32.800+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:32.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:32.803+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:32.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:32.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:32.804+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 98, above threshold 10. [2022-11-22T22:55:33.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 102889 ms [2022-11-22T22:55:33.850+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:33.851+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:33.853+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:33.854+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:33.854+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:33.854+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 99, above threshold 10. [2022-11-22T22:55:34.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 103939 ms [2022-11-22T22:55:34.901+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:34.901+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:34.904+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:34.905+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:34.905+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:34.905+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 100, above threshold 10. [2022-11-22T22:55:35.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 104989 ms [2022-11-22T22:55:35.950+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:35.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:35.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:35.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:35.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:35.954+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 101, above threshold 10. [2022-11-22T22:55:36.054+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:55:36.054+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:55:36.054+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:55:36.057+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:36.058+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:36.058+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:55:37.001+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 106040 ms [2022-11-22T22:55:37.001+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:37.001+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:37.004+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:37.004+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:37.004+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:37.004+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 102, above threshold 10. [2022-11-22T22:55:38.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 107089 ms [2022-11-22T22:55:38.050+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:38.051+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:38.053+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:38.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:38.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:38.054+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 103, above threshold 10. [2022-11-22T22:55:39.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 108139 ms [2022-11-22T22:55:39.100+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:39.101+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:39.104+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:39.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:39.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:39.104+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 104, above threshold 10. [2022-11-22T22:55:40.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 109189 ms [2022-11-22T22:55:40.150+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:40.150+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:40.153+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:40.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:40.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:40.153+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 105, above threshold 10. [2022-11-22T22:55:41.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 110239 ms [2022-11-22T22:55:41.200+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:41.200+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:41.203+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:41.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:41.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:41.204+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 106, above threshold 10. [2022-11-22T22:55:42.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 111289 ms [2022-11-22T22:55:42.250+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:42.251+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:42.254+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:42.255+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:42.255+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:42.255+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 107, above threshold 10. [2022-11-22T22:55:43.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 112339 ms [2022-11-22T22:55:43.300+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:43.301+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:43.303+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:43.305+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:43.305+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:43.305+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 108, above threshold 10. [2022-11-22T22:55:44.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 113389 ms [2022-11-22T22:55:44.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:44.351+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:44.353+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:44.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:44.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:44.354+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 109, above threshold 10. [2022-11-22T22:55:45.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 114439 ms [2022-11-22T22:55:45.400+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:45.401+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:45.403+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:45.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:45.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:45.403+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 110, above threshold 10. [2022-11-22T22:55:46.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 115489 ms [2022-11-22T22:55:46.450+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:46.451+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:46.456+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:46.457+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:46.457+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:46.457+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 111, above threshold 10. [2022-11-22T22:55:47.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 116539 ms [2022-11-22T22:55:47.500+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:47.501+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:47.503+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:47.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:47.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:47.503+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 112, above threshold 10. [2022-11-22T22:55:48.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 117589 ms [2022-11-22T22:55:48.550+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:48.551+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:48.553+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:48.553+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:48.553+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:48.553+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 113, above threshold 10. [2022-11-22T22:55:49.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 118639 ms [2022-11-22T22:55:49.600+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:49.600+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:49.603+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:49.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:49.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:49.603+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 114, above threshold 10. [2022-11-22T22:55:50.650+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 119689 ms [2022-11-22T22:55:50.651+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:50.651+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:50.654+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:50.654+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:50.654+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-11-22T22:55:50.654+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 115, above threshold 10. [2022-11-22T22:55:50.922+00:00|WARN|PdpMessageHandler|Timer-0] not registered: object:engine/apex/handler [2022-11-22T22:55:50.923+00:00|INFO|network|Timer-0] [OUT|DMAAP|POLICY-PDP-PAP] {"pdpType":"apex","state":"PASSIVE","healthy":"HEALTHY","description":"Pdp Heartbeat","statistics":{"pdpInstanceId":"apex-082c82bb-bec8-4e9d-9651-e46b70119585","timeStamp":"2022-11-22T22:55:50.922470Z","pdpGroupName":"defaultGroup","policyExecutedCount":0,"policyExecutedSuccessCount":0,"policyExecutedFailCount":0,"policyDeployCount":0,"policyDeploySuccessCount":0,"policyDeployFailCount":0,"policyUndeployCount":0,"policyUndeploySuccessCount":0,"policyUndeployFailCount":0},"messageName":"PDP_STATUS","requestId":"e0515131-ab9c-4b44-9548-e6c7dee93843","timestampMs":1669157750922,"name":"apex-082c82bb-bec8-4e9d-9651-e46b70119585","pdpGroup":"defaultGroup"} [2022-11-22T22:55:51.058+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:55:51.058+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:55:51.058+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:55:51.062+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:51.062+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:51.062+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:55:51.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 120739 ms [2022-11-22T22:55:51.701+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:51.701+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:51.704+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:51.704+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:51.705+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:55:51.705+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 116, above threshold 10. [2022-11-22T22:55:52.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 121789 ms [2022-11-22T22:55:52.751+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:52.752+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:52.756+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:52.757+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:52.757+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:55:52.757+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 117, above threshold 10. [2022-11-22T22:55:53.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 122839 ms [2022-11-22T22:55:53.800+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:53.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:53.803+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:53.803+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:53.803+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:55:53.803+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 118, above threshold 10. [2022-11-22T22:55:54.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 123889 ms [2022-11-22T22:55:54.850+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:54.851+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:54.854+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:54.854+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:54.854+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:55:54.854+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 119, above threshold 10. [2022-11-22T22:55:55.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 124939 ms [2022-11-22T22:55:55.901+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:55.901+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:55.906+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:55.907+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:55.908+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:55:55.908+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 120, above threshold 10. [2022-11-22T22:55:56.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 125989 ms [2022-11-22T22:55:56.951+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:56.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:56.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:56.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:56.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:55:56.954+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 121, above threshold 10. [2022-11-22T22:55:58.000+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 127039 ms [2022-11-22T22:55:58.001+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:58.001+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:58.004+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:58.005+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:58.005+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:55:58.005+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 122, above threshold 10. [2022-11-22T22:55:59.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 128089 ms [2022-11-22T22:55:59.051+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:55:59.051+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:55:59.054+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:55:59.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:55:59.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:55:59.054+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 123, above threshold 10. [2022-11-22T22:56:00.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 129139 ms [2022-11-22T22:56:00.100+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:00.101+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:00.104+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:00.105+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:00.105+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:00.105+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 124, above threshold 10. [2022-11-22T22:56:01.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 130189 ms [2022-11-22T22:56:01.151+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:01.151+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:01.154+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:01.154+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:01.154+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:01.154+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 125, above threshold 10. [2022-11-22T22:56:02.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 131239 ms [2022-11-22T22:56:02.200+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:02.201+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:02.203+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:02.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:02.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:02.204+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 126, above threshold 10. [2022-11-22T22:56:03.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 132289 ms [2022-11-22T22:56:03.250+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:03.250+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:03.253+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:03.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:03.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:03.253+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 127, above threshold 10. [2022-11-22T22:56:04.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 133339 ms [2022-11-22T22:56:04.300+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:04.301+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:04.303+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:04.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:04.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:04.303+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 128, above threshold 10. [2022-11-22T22:56:05.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 134389 ms [2022-11-22T22:56:05.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:05.351+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:05.353+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:05.353+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:05.353+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:05.353+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 129, above threshold 10. [2022-11-22T22:56:06.062+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:56:06.063+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:56:06.063+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:56:06.066+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:06.066+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:06.066+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:56:06.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 135439 ms [2022-11-22T22:56:06.401+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:06.401+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:06.403+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:06.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:06.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:06.403+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 130, above threshold 10. [2022-11-22T22:56:07.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 136489 ms [2022-11-22T22:56:07.450+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:07.451+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:07.453+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:07.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:07.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:07.453+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 131, above threshold 10. [2022-11-22T22:56:08.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 137539 ms [2022-11-22T22:56:08.500+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:08.501+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:08.503+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:08.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:08.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:08.503+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 132, above threshold 10. [2022-11-22T22:56:09.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 138589 ms [2022-11-22T22:56:09.551+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:09.551+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:09.553+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:09.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:09.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:09.554+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 133, above threshold 10. [2022-11-22T22:56:10.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 139639 ms [2022-11-22T22:56:10.601+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:10.601+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:10.603+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:10.604+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:10.604+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:10.604+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 134, above threshold 10. [2022-11-22T22:56:11.650+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 140689 ms [2022-11-22T22:56:11.650+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:11.651+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:11.653+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:11.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:11.654+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:11.654+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 135, above threshold 10. [2022-11-22T22:56:12.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 141739 ms [2022-11-22T22:56:12.700+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:12.701+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:12.703+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:12.704+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:12.704+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:12.704+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 136, above threshold 10. [2022-11-22T22:56:13.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 142789 ms [2022-11-22T22:56:13.750+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:13.751+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:13.753+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:13.754+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:13.754+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:13.754+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 137, above threshold 10. [2022-11-22T22:56:14.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 143839 ms [2022-11-22T22:56:14.800+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:14.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:14.803+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:14.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:14.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:14.804+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 138, above threshold 10. [2022-11-22T22:56:15.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 144889 ms [2022-11-22T22:56:15.850+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:15.851+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:15.853+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:15.853+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:15.854+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:15.854+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 139, above threshold 10. [2022-11-22T22:56:16.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 145939 ms [2022-11-22T22:56:16.900+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:16.901+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:16.903+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:16.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:16.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:16.904+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 140, above threshold 10. [2022-11-22T22:56:17.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 146989 ms [2022-11-22T22:56:17.950+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:17.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:17.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:17.953+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:17.953+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:17.953+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 141, above threshold 10. [2022-11-22T22:56:19.000+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 148039 ms [2022-11-22T22:56:19.000+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:19.000+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:19.003+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:19.003+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:19.003+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:19.003+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 142, above threshold 10. [2022-11-22T22:56:20.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 149089 ms [2022-11-22T22:56:20.051+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:20.051+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:20.053+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:20.053+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:20.054+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:20.054+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 143, above threshold 10. [2022-11-22T22:56:21.067+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:56:21.067+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:56:21.067+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:56:21.070+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:21.070+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:21.070+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:56:21.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 150139 ms [2022-11-22T22:56:21.100+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:21.101+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:21.103+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:21.103+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:21.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:21.104+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 144, above threshold 10. [2022-11-22T22:56:22.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 151189 ms [2022-11-22T22:56:22.151+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:22.151+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:22.153+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:22.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:22.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:22.153+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 145, above threshold 10. [2022-11-22T22:56:23.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 152239 ms [2022-11-22T22:56:23.201+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:23.201+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:23.205+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:23.206+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:23.206+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:23.206+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 146, above threshold 10. [2022-11-22T22:56:24.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 153289 ms [2022-11-22T22:56:24.251+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:24.251+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:24.254+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:24.255+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:24.255+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:24.255+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 147, above threshold 10. [2022-11-22T22:56:25.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 154339 ms [2022-11-22T22:56:25.301+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:25.301+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:25.304+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:25.304+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:25.304+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:25.305+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 148, above threshold 10. [2022-11-22T22:56:26.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 155389 ms [2022-11-22T22:56:26.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:26.350+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:26.352+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:26.353+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:26.353+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:26.353+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 149, above threshold 10. [2022-11-22T22:56:27.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 156439 ms [2022-11-22T22:56:27.400+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:27.400+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:27.403+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:27.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:27.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:27.403+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 150, above threshold 10. [2022-11-22T22:56:28.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 157489 ms [2022-11-22T22:56:28.450+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:28.451+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:28.453+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:28.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:28.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:28.453+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 151, above threshold 10. [2022-11-22T22:56:29.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 158539 ms [2022-11-22T22:56:29.500+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:29.500+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:29.503+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:29.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:29.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:29.503+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 152, above threshold 10. [2022-11-22T22:56:30.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 159589 ms [2022-11-22T22:56:30.550+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:30.550+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:30.553+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:30.553+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:30.553+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:30.553+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 153, above threshold 10. [2022-11-22T22:56:31.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 160639 ms [2022-11-22T22:56:31.601+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:31.601+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:31.603+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:31.604+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:31.604+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:31.604+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 154, above threshold 10. [2022-11-22T22:56:32.650+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 161689 ms [2022-11-22T22:56:32.651+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:32.651+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:32.654+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:32.654+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:32.654+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:32.654+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 155, above threshold 10. [2022-11-22T22:56:33.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 162739 ms [2022-11-22T22:56:33.700+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:33.701+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:33.703+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:33.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:33.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:33.703+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 156, above threshold 10. [2022-11-22T22:56:34.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 163789 ms [2022-11-22T22:56:34.751+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:34.751+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:34.753+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:34.754+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:34.754+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:34.754+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 157, above threshold 10. [2022-11-22T22:56:35.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 164839 ms [2022-11-22T22:56:35.801+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:35.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:35.804+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:35.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:35.805+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:35.805+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 158, above threshold 10. [2022-11-22T22:56:36.071+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:56:36.071+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:56:36.071+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:56:36.075+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:36.075+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:36.075+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:56:36.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 165889 ms [2022-11-22T22:56:36.851+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:36.851+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:36.854+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:36.855+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:36.855+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:36.855+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 159, above threshold 10. [2022-11-22T22:56:37.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 166939 ms [2022-11-22T22:56:37.901+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:37.901+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:37.903+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:37.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:37.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:37.904+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 160, above threshold 10. [2022-11-22T22:56:38.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 167989 ms [2022-11-22T22:56:38.950+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:38.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:38.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:38.953+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:38.953+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:38.954+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 161, above threshold 10. [2022-11-22T22:56:40.000+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 169039 ms [2022-11-22T22:56:40.000+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:40.001+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:40.003+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:40.004+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:40.004+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:40.004+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 162, above threshold 10. [2022-11-22T22:56:41.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 170089 ms [2022-11-22T22:56:41.050+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:41.051+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:41.053+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:41.053+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:41.053+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:41.054+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 163, above threshold 10. [2022-11-22T22:56:42.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 171139 ms [2022-11-22T22:56:42.100+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:42.100+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:42.102+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:42.103+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:42.103+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:42.103+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 164, above threshold 10. [2022-11-22T22:56:43.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 172189 ms [2022-11-22T22:56:43.150+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:43.150+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:43.153+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:43.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:43.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:43.153+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 165, above threshold 10. [2022-11-22T22:56:44.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 173239 ms [2022-11-22T22:56:44.200+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:44.200+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:44.202+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:44.203+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:44.203+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:44.203+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 166, above threshold 10. [2022-11-22T22:56:45.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 174289 ms [2022-11-22T22:56:45.250+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:45.250+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:45.252+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:45.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:45.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:45.253+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 167, above threshold 10. [2022-11-22T22:56:46.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 175339 ms [2022-11-22T22:56:46.300+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:46.300+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:46.302+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:46.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:46.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:46.303+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 168, above threshold 10. [2022-11-22T22:56:47.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 176389 ms [2022-11-22T22:56:47.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:47.351+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:47.353+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:47.353+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:47.353+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:47.353+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 169, above threshold 10. [2022-11-22T22:56:48.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 177439 ms [2022-11-22T22:56:48.400+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:48.401+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:48.403+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:48.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:48.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:48.403+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 170, above threshold 10. [2022-11-22T22:56:49.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 178489 ms [2022-11-22T22:56:49.450+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:49.450+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:49.453+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:49.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:49.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:49.453+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 171, above threshold 10. [2022-11-22T22:56:50.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 179539 ms [2022-11-22T22:56:50.500+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:50.500+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:50.502+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:50.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:50.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:50.503+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 172, above threshold 10. [2022-11-22T22:56:51.076+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:56:51.076+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:56:51.076+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:56:51.079+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:51.079+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:51.079+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:56:51.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 180589 ms [2022-11-22T22:56:51.550+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:51.551+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:51.553+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:51.553+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:51.553+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:51.553+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 173, above threshold 10. [2022-11-22T22:56:52.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 181639 ms [2022-11-22T22:56:52.600+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:52.601+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:52.603+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:52.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:52.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:52.603+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 174, above threshold 10. [2022-11-22T22:56:53.650+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 182689 ms [2022-11-22T22:56:53.650+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:53.651+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:53.655+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:53.655+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:53.655+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:53.655+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 175, above threshold 10. [2022-11-22T22:56:54.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 183739 ms [2022-11-22T22:56:54.700+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:54.701+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:54.703+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:54.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:54.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:54.703+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 176, above threshold 10. [2022-11-22T22:56:55.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 184789 ms [2022-11-22T22:56:55.750+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:55.751+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:55.752+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:55.753+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:55.753+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:55.753+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 177, above threshold 10. [2022-11-22T22:56:56.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 185839 ms [2022-11-22T22:56:56.800+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:56.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:56.804+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:56.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:56.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:56.804+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 178, above threshold 10. [2022-11-22T22:56:57.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 186889 ms [2022-11-22T22:56:57.850+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:57.851+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:57.853+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:57.853+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:57.853+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:57.853+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 179, above threshold 10. [2022-11-22T22:56:58.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 187939 ms [2022-11-22T22:56:58.900+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:58.901+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:58.903+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:58.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:58.904+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:58.904+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 180, above threshold 10. [2022-11-22T22:56:59.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 188989 ms [2022-11-22T22:56:59.950+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:56:59.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:56:59.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:56:59.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:56:59.954+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:56:59.954+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 181, above threshold 10. [2022-11-22T22:57:01.000+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 190039 ms [2022-11-22T22:57:01.000+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:01.000+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:01.002+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:01.003+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:01.003+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:01.003+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 182, above threshold 10. [2022-11-22T22:57:02.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 191089 ms [2022-11-22T22:57:02.050+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:02.050+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:02.053+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:02.053+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:02.053+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:02.053+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 183, above threshold 10. [2022-11-22T22:57:03.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 192139 ms [2022-11-22T22:57:03.100+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:03.100+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:03.103+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:03.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:03.104+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:03.104+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 184, above threshold 10. [2022-11-22T22:57:04.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 193189 ms [2022-11-22T22:57:04.150+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:04.150+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:04.153+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:04.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:04.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:04.153+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 185, above threshold 10. [2022-11-22T22:57:05.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 194239 ms [2022-11-22T22:57:05.200+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:05.201+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:05.203+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:05.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:05.204+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:05.204+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 186, above threshold 10. [2022-11-22T22:57:06.079+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:57:06.080+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:57:06.080+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:57:06.083+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:06.083+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:06.083+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:57:06.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 195289 ms [2022-11-22T22:57:06.250+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:06.251+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:06.253+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:06.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:06.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:06.253+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 187, above threshold 10. [2022-11-22T22:57:07.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 196339 ms [2022-11-22T22:57:07.300+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:07.300+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:07.302+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:07.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:07.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:07.303+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 188, above threshold 10. [2022-11-22T22:57:08.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 197389 ms [2022-11-22T22:57:08.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:08.351+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:08.353+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:08.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:08.354+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:08.354+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 189, above threshold 10. [2022-11-22T22:57:09.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 198439 ms [2022-11-22T22:57:09.400+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:09.400+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:09.403+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:09.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:09.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:09.403+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 190, above threshold 10. [2022-11-22T22:57:10.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 199489 ms [2022-11-22T22:57:10.450+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:10.451+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:10.453+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:10.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:10.453+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:10.453+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 191, above threshold 10. [2022-11-22T22:57:11.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 200539 ms [2022-11-22T22:57:11.500+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:11.500+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:11.502+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:11.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:11.503+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:11.503+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 192, above threshold 10. [2022-11-22T22:57:12.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 201589 ms [2022-11-22T22:57:12.550+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:12.551+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:12.554+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:12.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:12.554+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:12.554+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 193, above threshold 10. [2022-11-22T22:57:13.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 202639 ms [2022-11-22T22:57:13.600+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:13.601+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:13.603+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:13.604+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:13.604+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:13.604+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 194, above threshold 10. [2022-11-22T22:57:14.650+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 203689 ms [2022-11-22T22:57:14.650+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:14.651+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:14.653+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:14.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:14.654+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:14.654+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 195, above threshold 10. [2022-11-22T22:57:15.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 204739 ms [2022-11-22T22:57:15.701+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:15.701+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:15.705+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:15.705+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:15.705+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:15.705+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 196, above threshold 10. [2022-11-22T22:57:16.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 205789 ms [2022-11-22T22:57:16.750+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:16.750+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:16.752+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:16.753+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:16.753+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:16.753+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 197, above threshold 10. [2022-11-22T22:57:17.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 206839 ms [2022-11-22T22:57:17.800+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:17.800+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:17.803+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:17.803+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:17.803+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:17.803+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 198, above threshold 10. [2022-11-22T22:57:18.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 207889 ms [2022-11-22T22:57:18.851+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:18.851+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:18.854+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:18.855+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:18.855+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:18.855+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 199, above threshold 10. [2022-11-22T22:57:19.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 208939 ms [2022-11-22T22:57:19.900+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:19.901+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:19.903+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:19.905+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:19.905+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:19.905+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 200, above threshold 10. [2022-11-22T22:57:20.950+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 209989 ms [2022-11-22T22:57:20.950+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:20.951+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:20.953+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:20.953+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:20.953+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:20.953+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 201, above threshold 10. [2022-11-22T22:57:21.083+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:57:21.084+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:57:21.084+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:57:21.087+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:21.088+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:21.088+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:57:22.000+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 211039 ms [2022-11-22T22:57:22.000+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:22.000+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:22.003+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:22.003+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:22.003+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:22.003+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 202, above threshold 10. [2022-11-22T22:57:23.050+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 212089 ms [2022-11-22T22:57:23.050+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:23.050+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:23.053+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:23.053+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:23.053+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:23.053+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 203, above threshold 10. [2022-11-22T22:57:24.100+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 213139 ms [2022-11-22T22:57:24.100+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:24.101+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:24.106+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:24.107+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:24.107+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:24.107+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 204, above threshold 10. [2022-11-22T22:57:25.150+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 214189 ms [2022-11-22T22:57:25.150+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:25.151+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:25.153+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:25.153+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:25.154+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:25.154+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 205, above threshold 10. [2022-11-22T22:57:26.200+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 215239 ms [2022-11-22T22:57:26.200+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:26.200+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:26.203+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:26.203+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:26.203+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:26.203+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 206, above threshold 10. [2022-11-22T22:57:27.250+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 216289 ms [2022-11-22T22:57:27.250+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:27.250+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:27.252+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:27.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:27.253+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:27.253+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 207, above threshold 10. [2022-11-22T22:57:28.300+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 217339 ms [2022-11-22T22:57:28.300+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:28.301+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:28.303+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:28.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:28.303+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:28.304+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 208, above threshold 10. [2022-11-22T22:57:29.350+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 218389 ms [2022-11-22T22:57:29.350+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:29.350+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:29.353+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:29.353+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:29.359+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:29.359+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 209, above threshold 10. [2022-11-22T22:57:30.400+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 219439 ms [2022-11-22T22:57:30.400+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:30.401+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:30.403+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:30.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:30.403+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:30.403+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 210, above threshold 10. [2022-11-22T22:57:31.450+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 220489 ms [2022-11-22T22:57:31.450+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:31.451+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:31.454+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:31.455+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:31.455+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:31.455+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 211, above threshold 10. [2022-11-22T22:57:32.500+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 221539 ms [2022-11-22T22:57:32.501+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:32.501+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:32.504+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:32.504+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:32.504+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:32.504+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 212, above threshold 10. [2022-11-22T22:57:33.550+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 222589 ms [2022-11-22T22:57:33.550+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:33.551+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:33.559+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:33.560+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:33.560+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:33.560+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 213, above threshold 10. [2022-11-22T22:57:34.600+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 223639 ms [2022-11-22T22:57:34.600+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:34.601+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:34.603+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:34.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:34.603+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:34.603+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 214, above threshold 10. [2022-11-22T22:57:35.650+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 224689 ms [2022-11-22T22:57:35.650+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:35.650+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:35.653+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:35.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:35.653+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:35.653+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 215, above threshold 10. [2022-11-22T22:57:36.088+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=9f70185b-59c9-4726-aecb-565a20c5c994, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-11-22T22:57:36.089+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 [2022-11-22T22:57:36.089+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/9f70185b-59c9-4726-aecb-565a20c5c994/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-11-22T22:57:36.091+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:36.092+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:36.092+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-11-22T22:57:36.700+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 225739 ms [2022-11-22T22:57:36.700+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:36.700+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:36.702+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:36.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:36.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:36.703+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 216, above threshold 10. [2022-11-22T22:57:37.750+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 226789 ms [2022-11-22T22:57:37.750+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:37.751+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:37.753+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:37.753+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:37.753+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:37.753+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 217, above threshold 10. [2022-11-22T22:57:38.800+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 227839 ms [2022-11-22T22:57:38.800+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:38.801+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:38.803+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:38.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:38.804+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:38.804+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 218, above threshold 10. [2022-11-22T22:57:39.850+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 228889 ms [2022-11-22T22:57:39.850+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:39.851+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:39.853+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:39.853+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:39.853+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:39.853+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 219, above threshold 10. [2022-11-22T22:57:40.900+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 229939 ms [2022-11-22T22:57:40.900+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-11-22T22:57:40.900+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-11-22T22:57:40.903+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-11-22T22:57:40.903+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-22T22:57:40.903+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-11-22T22:57:40.903+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 220, above threshold 10. [2022