overriding logback.xml apexApps.sh: running application 'onappf' with command 'java -Dlogback.configurationFile=/opt/app/policy/apex-pdp/etc/logback.xml -cp /opt/app/policy/apex-pdp/etc:/opt/app/policy/apex-pdp/etc/hazelcast:/opt/app/policy/apex-pdp/etc/infinispan:/opt/app/policy/apex-pdp/lib/* -Djavax.net.ssl.keyStore=/opt/app/policy/apex-pdp/etc/ssl/policy-keystore -Djavax.net.ssl.keyStorePassword=Pol1cy_0nap -Djavax.net.ssl.trustStore=/opt/app/policy/apex-pdp/etc/ssl/policy-truststore -Djavax.net.ssl.trustStorePassword=Pol1cy_0nap -Dlogback.configurationFile=/opt/app/policy/apex-pdp/etc/logback.xml -Dhazelcast.config=/opt/app/policy/apex-pdp/etc/hazelcast.xml -Dhazelcast.mancenter.enabled=false org.onap.policy.apex.services.onappf.ApexStarterMain -c /home/apexuser/config/OnapPfConfig.json' [2023-01-13T23:06:57.803+00:00|INFO|ApexStarterMain|main] In ApexStarter with parameters [-c, /home/apexuser/config/OnapPfConfig.json] [2023-01-13T23:06:58.816+00:00|INFO|SingleThreadedDmaapTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: INITTED [2023-01-13T23:06:59.002+00:00|INFO|ServiceManager|main] service manager starting [2023-01-13T23:06:59.002+00:00|INFO|ServiceManager|main] service manager starting topics [2023-01-13T23:06:59.027+00:00|INFO|SingleThreadedBusTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: starting [2023-01-13T23:06:59.032+00:00|INFO|SingleThreadedDmaapTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: INITTED [2023-01-13T23:06:59.040+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:06:59.049+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:06:59.035+00:00|INFO|InlineBusTopicSink|main] InlineDmaapTopicSink [userName=null, password=null, getTopicCommInfrastructure()=DMAAP, toString()=InlineBusTopicSink [partitionId=8c6f4f2c-66db-4b62-a88a-4452bc8bb2d1, alive=false, publisher=null]]: starting [2023-01-13T23:06:59.076+00:00|INFO|InlineDmaapTopicSink|main] InlineDmaapTopicSink [userName=null, password=null, getTopicCommInfrastructure()=DMAAP, toString()=InlineBusTopicSink [partitionId=8c6f4f2c-66db-4b62-a88a-4452bc8bb2d1, alive=false, publisher=CambriaPublisherWrapper []]]: DMAAP SINK created [2023-01-13T23:06:59.076+00:00|INFO|ServiceManager|main] service manager starting set alive [2023-01-13T23:06:59.076+00:00|INFO|ServiceManager|main] service manager starting register pdp status context object [2023-01-13T23:06:59.081+00:00|INFO|ServiceManager|main] service manager starting topic sinks [2023-01-13T23:06:59.178+00:00|INFO|ServiceManager|main] service manager starting Pdp Status publisher [2023-01-13T23:06:59.184+00:00|INFO|ServiceManager|main] service manager starting Register pdp update listener [2023-01-13T23:06:59.184+00:00|INFO|ServiceManager|main] service manager starting Register pdp state change request dispatcher [2023-01-13T23:06:59.184+00:00|INFO|ServiceManager|main] service manager starting Message Dispatcher [2023-01-13T23:06:59.184+00:00|INFO|TopicBase|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: registering org.onap.policy.common.endpoints.listeners.MessageTypeDispatcher@4ebea12c [2023-01-13T23:06:59.185+00:00|INFO|SingleThreadedBusTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: register: start not attempted [2023-01-13T23:06:59.185+00:00|INFO|ServiceManager|main] service manager starting Create REST server [2023-01-13T23:06:59.186+00:00|WARN|PdpMessageHandler|Timer-0] not registered: object:engine/apex/handler [2023-01-13T23:06:59.220+00:00|INFO|OrderedServiceImpl|Timer-0] ***** OrderedServiceImpl implementers: [] [2023-01-13T23:06:59.224+00:00|INFO|network|Timer-0] [OUT|DMAAP|POLICY-PDP-PAP] {"pdpType":"apex","state":"PASSIVE","healthy":"HEALTHY","description":"Pdp Heartbeat","statistics":{"pdpInstanceId":"apex-ab64590f-9b97-45e9-9971-02d380d20256","timeStamp":"2023-01-13T23:06:59.189932Z","pdpGroupName":"defaultGroup","policyExecutedCount":0,"policyExecutedSuccessCount":0,"policyExecutedFailCount":0,"policyDeployCount":0,"policyDeploySuccessCount":0,"policyDeployFailCount":0,"policyUndeployCount":0,"policyUndeploySuccessCount":0,"policyUndeployFailCount":0},"messageName":"PDP_STATUS","requestId":"f630bf3c-6b15-4dec-8249-4f38cc9087ba","timestampMs":1673651219184,"name":"apex-ab64590f-9b97-45e9-9971-02d380d20256","pdpGroup":"defaultGroup"} [2023-01-13T23:06:59.276+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 48 ms [2023-01-13T23:06:59.279+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:06:59.298+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:06:59.298+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:06:59.304+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:06:59.304+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:06:59.311+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:06:59.311+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:06:59.318+00:00|INFO|log|main] Logging initialized @2572ms to org.eclipse.jetty.util.log.Slf4jLog [2023-01-13T23:06:59.562+00:00|INFO|ServiceManager|main] service manager starting Rest Server [2023-01-13T23:06:59.562+00:00|INFO|ServiceManager|main] service manager starting [2023-01-13T23:06:59.562+00:00|INFO|ServiceManager|main] service manager starting REST RestServerParameters [2023-01-13T23:06:59.563+00:00|INFO|JettyServletServer|main] JettyJerseyServer [Jerseyservlets={/metrics=io.prometheus.client.exporter.MetricsServlet-1e287667==io.prometheus.client.exporter.MetricsServlet@77636cbf{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-4d4d48a6==org.glassfish.jersey.servlet.ServletContainer@fafc7f67{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-3e2fc448==io.swagger.jersey.config.JerseyJaxrsConfig@e33c962e{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}}, swaggerId=swagger-6969, toString()=JettyServletServer(name=RestServerParameters, host=0.0.0.0, port=6969, user=healthcheck, password=zb!XztG34, contextPath=/, jettyServer=Server@33a053d{STOPPED}[9.4.48.v20220622], context=o.e.j.s.ServletContextHandler@14a54ef6{/,null,STOPPED}, connector=RestServerParameters@210386e0{HTTP/1.1, (http/1.1)}{0.0.0.0:6969}, jettyThread=null, servlets={/metrics=io.prometheus.client.exporter.MetricsServlet-1e287667==io.prometheus.client.exporter.MetricsServlet@77636cbf{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-4d4d48a6==org.glassfish.jersey.servlet.ServletContainer@fafc7f67{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-3e2fc448==io.swagger.jersey.config.JerseyJaxrsConfig@e33c962e{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}})]: STARTING [2023-01-13T23:06:59.612+00:00|INFO|ServiceManager|main] service manager started [2023-01-13T23:06:59.612+00:00|INFO|ServiceManager|main] service manager started [2023-01-13T23:06:59.612+00:00|INFO|ApexStarterMain|main] Started policy-apex-pdp service successfully. [2023-01-13T23:06:59.612+00:00|INFO|JettyServletServer|RestServerParameters-6969] JettyJerseyServer [Jerseyservlets={/metrics=io.prometheus.client.exporter.MetricsServlet-1e287667==io.prometheus.client.exporter.MetricsServlet@77636cbf{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-4d4d48a6==org.glassfish.jersey.servlet.ServletContainer@fafc7f67{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-3e2fc448==io.swagger.jersey.config.JerseyJaxrsConfig@e33c962e{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}}, swaggerId=swagger-6969, toString()=JettyServletServer(name=RestServerParameters, host=0.0.0.0, port=6969, user=healthcheck, password=zb!XztG34, contextPath=/, jettyServer=Server@33a053d{STOPPED}[9.4.48.v20220622], context=o.e.j.s.ServletContextHandler@14a54ef6{/,null,STOPPED}, connector=RestServerParameters@210386e0{HTTP/1.1, (http/1.1)}{0.0.0.0:6969}, jettyThread=Thread[RestServerParameters-6969,5,main], servlets={/metrics=io.prometheus.client.exporter.MetricsServlet-1e287667==io.prometheus.client.exporter.MetricsServlet@77636cbf{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-4d4d48a6==org.glassfish.jersey.servlet.ServletContainer@fafc7f67{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-3e2fc448==io.swagger.jersey.config.JerseyJaxrsConfig@e33c962e{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}})]: STARTING [2023-01-13T23:06:59.619+00:00|INFO|Server|RestServerParameters-6969] jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.15+10-alpine-r0 [2023-01-13T23:06:59.663+00:00|INFO|session|RestServerParameters-6969] DefaultSessionIdManager workerName=node0 [2023-01-13T23:06:59.664+00:00|INFO|session|RestServerParameters-6969] No SessionScavenger set, using defaults [2023-01-13T23:06:59.666+00:00|INFO|session|RestServerParameters-6969] node0 Scavenging every 600000ms [2023-01-13T23:07:00.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 1098 ms [2023-01-13T23:07:00.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:00.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:00.342+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:00.343+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:00.343+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.sun.xml.bind.v2.runtime.reflect.opt.Injector (file:/opt/app/policy/apex-pdp/lib/jaxb-impl-2.3.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) WARNING: Please consider reporting this to the maintainers of com.sun.xml.bind.v2.runtime.reflect.opt.Injector WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [2023-01-13T23:07:00.753+00:00|INFO|GsonMessageBodyHandler|RestServerParameters-6969] Using GSON for REST calls [2023-01-13T23:07:00.758+00:00|INFO|YamlMessageBodyHandler|RestServerParameters-6969] Accepting YAML for REST calls [2023-01-13T23:07:00.994+00:00|INFO|ContextHandler|RestServerParameters-6969] Started o.e.j.s.ServletContextHandler@14a54ef6{/,null,AVAILABLE} [2023-01-13T23:07:01.018+00:00|INFO|AbstractConnector|RestServerParameters-6969] Started RestServerParameters@210386e0{HTTP/1.1, (http/1.1)}{0.0.0.0:6969} [2023-01-13T23:07:01.019+00:00|INFO|Server|RestServerParameters-6969] Started @4273ms [2023-01-13T23:07:01.375+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 2148 ms [2023-01-13T23:07:01.376+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:01.376+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:01.379+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:01.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:01.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:02.426+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 3198 ms [2023-01-13T23:07:02.426+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:02.427+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:02.430+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:02.431+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:02.432+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:03.476+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 4248 ms [2023-01-13T23:07:03.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:03.477+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:03.480+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:03.481+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:03.481+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:04.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 5298 ms [2023-01-13T23:07:04.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:04.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:04.529+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:04.530+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:04.530+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:05.576+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 6349 ms [2023-01-13T23:07:05.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:05.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:05.579+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:05.580+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:05.580+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:06.625+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 7398 ms [2023-01-13T23:07:06.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:06.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:06.628+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:06.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:06.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:07.676+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 8448 ms [2023-01-13T23:07:07.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:07.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:07.679+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:07.680+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:07.680+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:08.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 9498 ms [2023-01-13T23:07:08.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:08.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:08.728+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:08.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:08.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:09.777+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 10550 ms [2023-01-13T23:07:09.777+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:09.777+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:09.785+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:09.785+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:09.785+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:09.785+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 11, above threshold 10. [2023-01-13T23:07:10.829+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 11602 ms [2023-01-13T23:07:10.829+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:10.829+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:10.846+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:10.847+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:10.847+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:10.847+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 12, above threshold 10. [2023-01-13T23:07:11.875+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 12648 ms [2023-01-13T23:07:11.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:11.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:11.880+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:11.881+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:11.881+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:11.881+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 13, above threshold 10. [2023-01-13T23:07:12.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 13698 ms [2023-01-13T23:07:12.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:12.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:12.928+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:12.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:12.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:12.929+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 14, above threshold 10. [2023-01-13T23:07:13.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 14748 ms [2023-01-13T23:07:13.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:13.977+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:13.982+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:13.983+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:13.983+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:13.983+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 15, above threshold 10. [2023-01-13T23:07:14.306+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:07:14.306+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:07:14.306+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:07:14.310+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:14.310+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:14.310+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:07:15.025+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 15798 ms [2023-01-13T23:07:15.026+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:15.027+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:15.033+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:15.039+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:15.039+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:15.039+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 16, above threshold 10. [2023-01-13T23:07:16.075+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 16848 ms [2023-01-13T23:07:16.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:16.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:16.079+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:16.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:16.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:16.079+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 17, above threshold 10. [2023-01-13T23:07:17.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 17898 ms [2023-01-13T23:07:17.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:17.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:17.129+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:17.130+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:17.130+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:17.130+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 18, above threshold 10. [2023-01-13T23:07:18.175+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 18948 ms [2023-01-13T23:07:18.176+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:18.177+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:18.182+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:18.182+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:18.182+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:18.182+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 19, above threshold 10. [2023-01-13T23:07:19.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 19998 ms [2023-01-13T23:07:19.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:19.226+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:19.229+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:19.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:19.231+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:19.231+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 20, above threshold 10. [2023-01-13T23:07:20.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 21048 ms [2023-01-13T23:07:20.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:20.277+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:20.292+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:20.294+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:20.294+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:20.294+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 21, above threshold 10. [2023-01-13T23:07:21.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 22098 ms [2023-01-13T23:07:21.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:21.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:21.337+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:21.337+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:21.338+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:21.338+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 22, above threshold 10. [2023-01-13T23:07:22.375+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 23148 ms [2023-01-13T23:07:22.376+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:22.376+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:22.378+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:22.378+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:22.378+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:22.379+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23, above threshold 10. [2023-01-13T23:07:23.425+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 24198 ms [2023-01-13T23:07:23.426+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:23.426+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:23.428+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:23.429+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:23.429+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:23.429+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 24, above threshold 10. [2023-01-13T23:07:24.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 25248 ms [2023-01-13T23:07:24.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:24.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:24.478+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:24.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:24.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:24.479+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 25, above threshold 10. [2023-01-13T23:07:25.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 26298 ms [2023-01-13T23:07:25.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:25.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:25.529+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:25.531+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:25.531+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:25.531+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 26, above threshold 10. [2023-01-13T23:07:26.575+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 27348 ms [2023-01-13T23:07:26.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:26.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:26.578+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:26.578+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:26.578+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:26.578+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 27, above threshold 10. [2023-01-13T23:07:27.625+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 28398 ms [2023-01-13T23:07:27.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:27.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:27.633+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:27.633+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:27.633+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:27.633+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 28, above threshold 10. [2023-01-13T23:07:28.675+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 29448 ms [2023-01-13T23:07:28.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:28.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:28.678+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:28.678+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:28.678+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:28.678+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 29, above threshold 10. [2023-01-13T23:07:29.311+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:07:29.311+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:07:29.311+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:07:29.314+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:29.314+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:29.314+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:07:29.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 30498 ms [2023-01-13T23:07:29.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:29.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:29.737+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:29.737+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:29.738+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:29.738+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 30, above threshold 10. [2023-01-13T23:07:30.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 31548 ms [2023-01-13T23:07:30.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:30.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:30.778+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:30.778+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:30.778+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:30.778+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 31, above threshold 10. [2023-01-13T23:07:31.825+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 32598 ms [2023-01-13T23:07:31.826+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:31.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:31.828+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:31.828+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:31.828+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:31.828+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 32, above threshold 10. [2023-01-13T23:07:32.875+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 33648 ms [2023-01-13T23:07:32.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:32.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:32.878+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:32.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:32.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:32.879+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 33, above threshold 10. [2023-01-13T23:07:33.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 34698 ms [2023-01-13T23:07:33.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:33.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:33.928+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:33.928+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:33.928+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:33.929+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 34, above threshold 10. [2023-01-13T23:07:34.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 35748 ms [2023-01-13T23:07:34.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:34.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:34.979+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:34.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:34.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:34.980+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 35, above threshold 10. [2023-01-13T23:07:36.026+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 36798 ms [2023-01-13T23:07:36.026+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:36.026+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:36.033+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:36.035+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:36.035+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:36.035+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 36, above threshold 10. [2023-01-13T23:07:37.077+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 37850 ms [2023-01-13T23:07:37.077+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:37.077+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:37.080+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:37.081+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:37.081+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:37.081+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 37, above threshold 10. [2023-01-13T23:07:38.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 38898 ms [2023-01-13T23:07:38.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:38.127+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:38.129+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:38.133+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:38.133+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:38.137+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 38, above threshold 10. [2023-01-13T23:07:39.175+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 39948 ms [2023-01-13T23:07:39.176+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:39.176+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:39.178+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:39.178+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:39.179+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:39.179+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 39, above threshold 10. [2023-01-13T23:07:40.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 40998 ms [2023-01-13T23:07:40.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:40.226+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:40.229+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:40.238+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:40.238+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:40.238+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 40, above threshold 10. [2023-01-13T23:07:41.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 42048 ms [2023-01-13T23:07:41.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:41.276+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:41.279+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:41.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:41.280+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:41.280+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 41, above threshold 10. [2023-01-13T23:07:42.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 43098 ms [2023-01-13T23:07:42.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:42.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:42.329+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:42.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:42.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:42.329+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 42, above threshold 10. [2023-01-13T23:07:43.375+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 44148 ms [2023-01-13T23:07:43.376+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:43.376+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:43.378+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:43.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:43.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:43.379+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 43, above threshold 10. [2023-01-13T23:07:44.315+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:07:44.315+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:07:44.315+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:07:44.318+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:44.319+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:44.319+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:07:44.425+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 45198 ms [2023-01-13T23:07:44.426+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:44.426+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:44.428+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:44.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:44.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:44.428+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 44, above threshold 10. [2023-01-13T23:07:45.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 46248 ms [2023-01-13T23:07:45.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:45.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:45.478+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:45.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:45.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:45.479+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 45, above threshold 10. [2023-01-13T23:07:46.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 47298 ms [2023-01-13T23:07:46.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:46.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:46.529+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:46.529+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:46.530+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:46.530+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 46, above threshold 10. [2023-01-13T23:07:47.575+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 48348 ms [2023-01-13T23:07:47.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:47.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:47.579+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:47.579+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:47.579+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:47.579+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 47, above threshold 10. [2023-01-13T23:07:48.625+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 49398 ms [2023-01-13T23:07:48.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:48.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:48.629+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:48.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:48.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:48.629+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 48, above threshold 10. [2023-01-13T23:07:49.675+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 50448 ms [2023-01-13T23:07:49.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:49.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:49.678+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:49.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:49.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:49.679+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 49, above threshold 10. [2023-01-13T23:07:50.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 51498 ms [2023-01-13T23:07:50.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:50.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:50.729+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:50.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:50.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:50.729+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 50, above threshold 10. [2023-01-13T23:07:51.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 52548 ms [2023-01-13T23:07:51.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:51.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:51.778+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:51.778+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:51.778+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:51.778+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 51, above threshold 10. [2023-01-13T23:07:52.825+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 53598 ms [2023-01-13T23:07:52.826+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:52.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:52.828+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:52.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:52.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:52.829+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 52, above threshold 10. [2023-01-13T23:07:53.875+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 54648 ms [2023-01-13T23:07:53.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:53.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:53.878+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:53.878+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:53.878+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:53.878+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 53, above threshold 10. [2023-01-13T23:07:54.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 55698 ms [2023-01-13T23:07:54.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:54.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:54.928+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:54.928+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:54.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:54.929+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 54, above threshold 10. [2023-01-13T23:07:55.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 56748 ms [2023-01-13T23:07:55.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:55.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:55.978+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:55.978+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:55.978+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:55.978+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 55, above threshold 10. [2023-01-13T23:07:57.025+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 57798 ms [2023-01-13T23:07:57.026+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:57.026+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:57.028+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:57.028+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:57.029+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:57.029+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 56, above threshold 10. [2023-01-13T23:07:58.076+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 58848 ms [2023-01-13T23:07:58.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:58.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:58.079+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:58.080+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:58.080+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:58.080+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 57, above threshold 10. [2023-01-13T23:07:59.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 59898 ms [2023-01-13T23:07:59.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:07:59.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:07:59.129+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:59.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:07:59.130+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:07:59.130+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 58, above threshold 10. [2023-01-13T23:07:59.319+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:07:59.319+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:07:59.320+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:07:59.323+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:59.323+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:07:59.323+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:08:00.175+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 60948 ms [2023-01-13T23:08:00.176+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:00.177+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:00.181+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:00.182+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:00.183+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:00.184+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 59, above threshold 10. [2023-01-13T23:08:01.226+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 61998 ms [2023-01-13T23:08:01.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:01.227+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:01.230+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:01.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:01.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:01.230+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 60, above threshold 10. [2023-01-13T23:08:02.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 63048 ms [2023-01-13T23:08:02.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:02.276+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:02.278+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:02.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:02.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:02.279+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 61, above threshold 10. [2023-01-13T23:08:03.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 64098 ms [2023-01-13T23:08:03.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:03.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:03.328+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:03.328+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:03.328+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:03.328+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 62, above threshold 10. [2023-01-13T23:08:04.375+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 65148 ms [2023-01-13T23:08:04.376+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:04.376+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:04.379+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:04.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:04.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:04.379+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 63, above threshold 10. [2023-01-13T23:08:05.425+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 66198 ms [2023-01-13T23:08:05.426+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:05.426+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:05.428+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:05.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:05.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:05.428+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 64, above threshold 10. [2023-01-13T23:08:06.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 67248 ms [2023-01-13T23:08:06.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:06.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:06.478+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:06.478+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:06.478+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:06.478+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 65, above threshold 10. [2023-01-13T23:08:07.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 68298 ms [2023-01-13T23:08:07.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:07.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:07.528+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:07.528+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:07.528+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:07.528+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 66, above threshold 10. [2023-01-13T23:08:08.575+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 69348 ms [2023-01-13T23:08:08.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:08.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:08.578+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:08.578+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:08.578+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:08.578+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 67, above threshold 10. [2023-01-13T23:08:09.625+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 70398 ms [2023-01-13T23:08:09.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:09.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:09.629+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:09.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:09.630+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:09.630+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 68, above threshold 10. [2023-01-13T23:08:10.675+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 71448 ms [2023-01-13T23:08:10.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:10.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:10.678+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:10.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:10.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:10.679+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 69, above threshold 10. [2023-01-13T23:08:11.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 72498 ms [2023-01-13T23:08:11.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:11.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:11.728+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:11.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:11.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:11.729+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 70, above threshold 10. [2023-01-13T23:08:12.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 73548 ms [2023-01-13T23:08:12.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:12.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:12.779+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:12.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:12.780+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:12.780+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 71, above threshold 10. [2023-01-13T23:08:13.825+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 74598 ms [2023-01-13T23:08:13.826+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:13.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:13.828+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:13.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:13.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:13.829+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 72, above threshold 10. [2023-01-13T23:08:14.324+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:08:14.324+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:08:14.324+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:08:14.327+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:14.327+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:14.328+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:08:14.875+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 75648 ms [2023-01-13T23:08:14.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:14.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:14.878+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:14.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:14.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:14.879+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 73, above threshold 10. [2023-01-13T23:08:15.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 76698 ms [2023-01-13T23:08:15.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:15.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:15.928+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:15.928+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:15.928+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:15.928+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 74, above threshold 10. [2023-01-13T23:08:16.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 77748 ms [2023-01-13T23:08:16.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:16.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:16.979+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:16.979+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:16.979+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:16.980+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 75, above threshold 10. [2023-01-13T23:08:18.025+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 78798 ms [2023-01-13T23:08:18.026+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:18.026+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:18.037+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:18.037+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:18.037+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:18.037+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 76, above threshold 10. [2023-01-13T23:08:19.076+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 79848 ms [2023-01-13T23:08:19.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:19.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:19.078+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:19.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:19.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:19.079+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 77, above threshold 10. [2023-01-13T23:08:20.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 80898 ms [2023-01-13T23:08:20.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:20.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:20.129+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:20.133+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:20.133+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:20.133+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 78, above threshold 10. [2023-01-13T23:08:21.177+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 81949 ms [2023-01-13T23:08:21.177+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:21.180+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:21.184+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:21.185+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:21.185+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:21.185+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 79, above threshold 10. [2023-01-13T23:08:22.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 82998 ms [2023-01-13T23:08:22.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:22.226+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:22.229+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:22.229+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:22.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:22.230+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 80, above threshold 10. [2023-01-13T23:08:23.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 84048 ms [2023-01-13T23:08:23.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:23.276+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:23.279+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:23.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:23.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:23.279+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 81, above threshold 10. [2023-01-13T23:08:24.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 85098 ms [2023-01-13T23:08:24.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:24.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:24.328+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:24.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:24.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:24.329+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 82, above threshold 10. [2023-01-13T23:08:25.375+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 86148 ms [2023-01-13T23:08:25.376+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:25.376+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:25.379+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:25.380+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:25.380+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:25.380+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 83, above threshold 10. [2023-01-13T23:08:26.425+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 87198 ms [2023-01-13T23:08:26.426+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:26.426+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:26.428+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:26.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:26.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:26.428+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 84, above threshold 10. [2023-01-13T23:08:27.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 88248 ms [2023-01-13T23:08:27.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:27.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:27.478+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:27.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:27.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:27.479+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 85, above threshold 10. [2023-01-13T23:08:28.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 89298 ms [2023-01-13T23:08:28.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:28.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:28.528+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:28.528+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:28.528+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:28.528+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 86, above threshold 10. [2023-01-13T23:08:29.328+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:08:29.329+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:08:29.329+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:08:29.332+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:29.332+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:29.332+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:08:29.576+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 90349 ms [2023-01-13T23:08:29.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:29.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:29.579+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:29.579+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:29.579+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:29.579+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 87, above threshold 10. [2023-01-13T23:08:30.625+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 91398 ms [2023-01-13T23:08:30.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:30.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:30.630+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:30.630+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:30.630+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:30.630+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 88, above threshold 10. [2023-01-13T23:08:31.675+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 92448 ms [2023-01-13T23:08:31.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:31.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:31.678+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:31.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:31.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:31.679+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 89, above threshold 10. [2023-01-13T23:08:32.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 93498 ms [2023-01-13T23:08:32.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:32.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:32.728+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:32.728+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:32.728+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:32.728+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 90, above threshold 10. [2023-01-13T23:08:33.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 94548 ms [2023-01-13T23:08:33.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:33.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:33.778+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:33.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:33.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:33.779+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 91, above threshold 10. [2023-01-13T23:08:34.826+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 95598 ms [2023-01-13T23:08:34.826+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:34.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:34.828+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:34.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:34.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:34.829+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 92, above threshold 10. [2023-01-13T23:08:35.875+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 96648 ms [2023-01-13T23:08:35.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:35.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:35.878+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:35.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:35.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:35.879+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 93, above threshold 10. [2023-01-13T23:08:36.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 97698 ms [2023-01-13T23:08:36.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:36.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:36.929+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:36.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:36.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:36.929+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 94, above threshold 10. [2023-01-13T23:08:37.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 98748 ms [2023-01-13T23:08:37.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:37.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:37.979+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:37.979+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:37.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:37.980+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 95, above threshold 10. [2023-01-13T23:08:39.025+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 99798 ms [2023-01-13T23:08:39.028+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:39.028+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:39.037+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:39.041+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:39.041+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:39.041+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 96, above threshold 10. [2023-01-13T23:08:40.075+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 100848 ms [2023-01-13T23:08:40.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:40.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:40.078+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:40.078+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:40.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:40.079+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 97, above threshold 10. [2023-01-13T23:08:41.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 101898 ms [2023-01-13T23:08:41.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:41.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:41.128+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:41.128+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:41.128+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:41.128+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 98, above threshold 10. [2023-01-13T23:08:42.175+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 102948 ms [2023-01-13T23:08:42.176+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:42.176+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:42.178+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:42.178+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:42.179+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:42.179+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 99, above threshold 10. [2023-01-13T23:08:43.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 103998 ms [2023-01-13T23:08:43.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:43.226+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:43.228+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:43.229+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:43.229+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:43.229+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 100, above threshold 10. [2023-01-13T23:08:44.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 105048 ms [2023-01-13T23:08:44.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:44.276+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:44.278+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:44.278+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:44.278+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:44.278+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 101, above threshold 10. [2023-01-13T23:08:44.333+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:08:44.333+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:08:44.333+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:08:44.338+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:44.338+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:44.338+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:08:45.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 106098 ms [2023-01-13T23:08:45.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:45.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:45.328+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:45.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:45.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:45.329+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 102, above threshold 10. [2023-01-13T23:08:46.377+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 107150 ms [2023-01-13T23:08:46.377+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:46.377+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:46.380+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:46.380+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:46.380+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:46.380+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 103, above threshold 10. [2023-01-13T23:08:47.425+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 108198 ms [2023-01-13T23:08:47.426+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:47.426+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:47.428+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:47.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:47.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:47.428+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 104, above threshold 10. [2023-01-13T23:08:48.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 109248 ms [2023-01-13T23:08:48.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:48.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:48.478+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:48.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:48.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:48.479+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 105, above threshold 10. [2023-01-13T23:08:49.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 110298 ms [2023-01-13T23:08:49.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:49.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:49.528+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:49.529+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:49.529+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:49.529+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 106, above threshold 10. [2023-01-13T23:08:50.575+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 111348 ms [2023-01-13T23:08:50.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:50.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:50.578+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:50.578+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:50.578+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:50.578+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 107, above threshold 10. [2023-01-13T23:08:51.625+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 112398 ms [2023-01-13T23:08:51.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:51.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:51.628+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:51.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:51.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:51.629+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 108, above threshold 10. [2023-01-13T23:08:52.676+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 113448 ms [2023-01-13T23:08:52.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:52.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:52.679+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:52.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:52.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:52.679+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 109, above threshold 10. [2023-01-13T23:08:53.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 114498 ms [2023-01-13T23:08:53.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:53.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:53.728+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:53.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:53.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:53.729+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 110, above threshold 10. [2023-01-13T23:08:54.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 115548 ms [2023-01-13T23:08:54.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:54.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:54.781+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:54.781+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:54.781+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:54.781+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 111, above threshold 10. [2023-01-13T23:08:55.825+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 116598 ms [2023-01-13T23:08:55.826+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:55.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:55.828+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:55.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:55.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:55.829+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 112, above threshold 10. [2023-01-13T23:08:56.876+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 117648 ms [2023-01-13T23:08:56.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:56.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:56.878+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:56.878+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:56.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:56.879+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 113, above threshold 10. [2023-01-13T23:08:57.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 118698 ms [2023-01-13T23:08:57.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:57.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:57.928+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:57.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:57.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:57.929+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 114, above threshold 10. [2023-01-13T23:08:58.976+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 119748 ms [2023-01-13T23:08:58.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:08:58.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:08:58.978+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:58.979+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:08:58.979+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2023-01-13T23:08:58.979+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 115, above threshold 10. [2023-01-13T23:08:59.184+00:00|WARN|PdpMessageHandler|Timer-0] not registered: object:engine/apex/handler [2023-01-13T23:08:59.185+00:00|INFO|network|Timer-0] [OUT|DMAAP|POLICY-PDP-PAP] {"pdpType":"apex","state":"PASSIVE","healthy":"HEALTHY","description":"Pdp Heartbeat","statistics":{"pdpInstanceId":"apex-ab64590f-9b97-45e9-9971-02d380d20256","timeStamp":"2023-01-13T23:08:59.184698Z","pdpGroupName":"defaultGroup","policyExecutedCount":0,"policyExecutedSuccessCount":0,"policyExecutedFailCount":0,"policyDeployCount":0,"policyDeploySuccessCount":0,"policyDeployFailCount":0,"policyUndeployCount":0,"policyUndeploySuccessCount":0,"policyUndeployFailCount":0},"messageName":"PDP_STATUS","requestId":"62c5be81-83e3-44c8-ad30-ec4cf3518f05","timestampMs":1673651339184,"name":"apex-ab64590f-9b97-45e9-9971-02d380d20256","pdpGroup":"defaultGroup"} [2023-01-13T23:08:59.338+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:08:59.339+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:08:59.339+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:08:59.342+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:59.342+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:08:59.342+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:09:00.025+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 120798 ms [2023-01-13T23:09:00.026+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:00.026+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:00.030+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:00.030+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:00.030+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:00.031+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 116, above threshold 10. [2023-01-13T23:09:01.075+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 121848 ms [2023-01-13T23:09:01.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:01.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:01.081+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:01.081+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:01.082+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:01.082+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 117, above threshold 10. [2023-01-13T23:09:02.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 122898 ms [2023-01-13T23:09:02.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:02.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:02.128+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:02.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:02.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:02.130+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 118, above threshold 10. [2023-01-13T23:09:03.175+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 123948 ms [2023-01-13T23:09:03.176+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:03.176+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:03.178+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:03.179+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:03.179+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:03.179+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 119, above threshold 10. [2023-01-13T23:09:04.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 124998 ms [2023-01-13T23:09:04.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:04.226+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:04.229+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:04.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:04.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:04.230+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 120, above threshold 10. [2023-01-13T23:09:05.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 126048 ms [2023-01-13T23:09:05.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:05.276+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:05.279+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:05.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:05.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:05.279+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 121, above threshold 10. [2023-01-13T23:09:06.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 127098 ms [2023-01-13T23:09:06.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:06.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:06.329+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:06.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:06.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:06.330+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 122, above threshold 10. [2023-01-13T23:09:07.375+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 128148 ms [2023-01-13T23:09:07.376+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:07.376+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:07.379+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:07.381+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:07.381+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:07.381+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 123, above threshold 10. [2023-01-13T23:09:08.425+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 129198 ms [2023-01-13T23:09:08.426+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:08.426+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:08.429+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:08.434+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:08.434+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:08.434+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 124, above threshold 10. [2023-01-13T23:09:09.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 130248 ms [2023-01-13T23:09:09.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:09.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:09.479+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:09.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:09.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:09.479+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 125, above threshold 10. [2023-01-13T23:09:10.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 131298 ms [2023-01-13T23:09:10.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:10.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:10.533+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:10.533+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:10.534+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:10.534+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 126, above threshold 10. [2023-01-13T23:09:11.575+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 132348 ms [2023-01-13T23:09:11.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:11.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:11.579+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:11.579+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:11.579+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:11.580+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 127, above threshold 10. [2023-01-13T23:09:12.625+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 133398 ms [2023-01-13T23:09:12.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:12.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:12.629+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:12.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:12.630+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:12.630+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 128, above threshold 10. [2023-01-13T23:09:13.675+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 134448 ms [2023-01-13T23:09:13.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:13.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:13.679+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:13.680+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:13.680+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:13.680+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 129, above threshold 10. [2023-01-13T23:09:14.343+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:09:14.343+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:09:14.343+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:09:14.346+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:14.346+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:14.346+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:09:14.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 135498 ms [2023-01-13T23:09:14.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:14.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:14.728+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:14.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:14.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:14.729+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 130, above threshold 10. [2023-01-13T23:09:15.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 136548 ms [2023-01-13T23:09:15.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:15.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:15.778+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:15.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:15.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:15.779+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 131, above threshold 10. [2023-01-13T23:09:16.825+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 137598 ms [2023-01-13T23:09:16.825+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:16.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:16.828+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:16.828+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:16.828+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:16.828+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 132, above threshold 10. [2023-01-13T23:09:17.875+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 138648 ms [2023-01-13T23:09:17.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:17.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:17.879+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:17.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:17.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:17.879+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 133, above threshold 10. [2023-01-13T23:09:18.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 139698 ms [2023-01-13T23:09:18.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:18.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:18.928+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:18.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:18.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:18.929+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 134, above threshold 10. [2023-01-13T23:09:19.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 140748 ms [2023-01-13T23:09:19.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:19.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:19.978+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:19.979+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:19.979+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:19.979+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 135, above threshold 10. [2023-01-13T23:09:21.025+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 141798 ms [2023-01-13T23:09:21.026+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:21.026+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:21.028+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:21.029+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:21.029+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:21.033+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 136, above threshold 10. [2023-01-13T23:09:22.075+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 142848 ms [2023-01-13T23:09:22.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:22.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:22.079+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:22.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:22.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:22.080+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 137, above threshold 10. [2023-01-13T23:09:23.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 143898 ms [2023-01-13T23:09:23.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:23.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:23.128+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:23.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:23.132+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:23.132+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 138, above threshold 10. [2023-01-13T23:09:24.175+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 144948 ms [2023-01-13T23:09:24.176+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:24.176+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:24.178+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:24.179+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:24.179+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:24.179+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 139, above threshold 10. [2023-01-13T23:09:25.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 145998 ms [2023-01-13T23:09:25.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:25.226+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:25.229+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:25.229+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:25.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:25.230+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 140, above threshold 10. [2023-01-13T23:09:26.276+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 147049 ms [2023-01-13T23:09:26.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:26.276+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:26.279+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:26.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:26.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:26.279+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 141, above threshold 10. [2023-01-13T23:09:27.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 148098 ms [2023-01-13T23:09:27.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:27.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:27.328+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:27.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:27.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:27.329+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 142, above threshold 10. [2023-01-13T23:09:28.375+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 149148 ms [2023-01-13T23:09:28.376+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:28.376+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:28.378+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:28.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:28.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:28.379+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 143, above threshold 10. [2023-01-13T23:09:29.347+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:09:29.347+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:09:29.347+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:09:29.350+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:29.350+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:29.350+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:09:29.425+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 150198 ms [2023-01-13T23:09:29.426+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:29.426+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:29.428+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:29.429+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:29.429+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:29.429+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 144, above threshold 10. [2023-01-13T23:09:30.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 151248 ms [2023-01-13T23:09:30.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:30.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:30.478+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:30.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:30.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:30.479+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 145, above threshold 10. [2023-01-13T23:09:31.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 152298 ms [2023-01-13T23:09:31.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:31.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:31.531+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:31.531+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:31.531+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:31.531+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 146, above threshold 10. [2023-01-13T23:09:32.575+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 153348 ms [2023-01-13T23:09:32.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:32.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:32.578+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:32.579+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:32.579+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:32.579+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 147, above threshold 10. [2023-01-13T23:09:33.626+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 154398 ms [2023-01-13T23:09:33.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:33.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:33.628+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:33.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:33.629+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:33.629+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 148, above threshold 10. [2023-01-13T23:09:34.675+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 155448 ms [2023-01-13T23:09:34.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:34.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:34.678+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:34.678+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:34.678+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:34.678+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 149, above threshold 10. [2023-01-13T23:09:35.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 156498 ms [2023-01-13T23:09:35.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:35.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:35.728+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:35.728+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:35.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:35.729+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 150, above threshold 10. [2023-01-13T23:09:36.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 157548 ms [2023-01-13T23:09:36.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:36.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:36.779+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:36.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:36.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:36.779+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 151, above threshold 10. [2023-01-13T23:09:37.825+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 158598 ms [2023-01-13T23:09:37.826+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:37.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:37.829+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:37.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:37.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:37.829+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 152, above threshold 10. [2023-01-13T23:09:38.875+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 159648 ms [2023-01-13T23:09:38.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:38.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:38.878+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:38.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:38.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:38.879+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 153, above threshold 10. [2023-01-13T23:09:39.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 160698 ms [2023-01-13T23:09:39.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:39.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:39.929+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:39.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:39.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:39.929+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 154, above threshold 10. [2023-01-13T23:09:40.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 161748 ms [2023-01-13T23:09:40.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:40.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:40.978+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:40.979+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:40.979+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:40.979+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 155, above threshold 10. [2023-01-13T23:09:42.025+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 162798 ms [2023-01-13T23:09:42.026+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:42.026+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:42.028+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:42.028+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:42.028+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:42.028+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 156, above threshold 10. [2023-01-13T23:09:43.075+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 163848 ms [2023-01-13T23:09:43.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:43.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:43.078+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:43.078+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:43.078+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:43.078+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 157, above threshold 10. [2023-01-13T23:09:44.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 164898 ms [2023-01-13T23:09:44.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:44.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:44.128+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:44.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:44.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:44.129+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 158, above threshold 10. [2023-01-13T23:09:44.351+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:09:44.351+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:09:44.351+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:09:44.355+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:44.355+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:44.355+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:09:45.175+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 165948 ms [2023-01-13T23:09:45.176+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:45.176+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:45.179+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:45.180+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:45.180+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:45.180+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 159, above threshold 10. [2023-01-13T23:09:46.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 166998 ms [2023-01-13T23:09:46.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:46.226+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:46.229+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:46.229+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:46.240+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:46.242+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 160, above threshold 10. [2023-01-13T23:09:47.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 168048 ms [2023-01-13T23:09:47.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:47.276+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:47.279+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:47.279+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:47.280+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:47.280+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 161, above threshold 10. [2023-01-13T23:09:48.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 169098 ms [2023-01-13T23:09:48.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:48.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:48.329+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:48.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:48.330+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:48.330+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 162, above threshold 10. [2023-01-13T23:09:49.377+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 170150 ms [2023-01-13T23:09:49.377+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:49.377+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:49.380+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:49.380+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:49.381+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:49.381+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 163, above threshold 10. [2023-01-13T23:09:50.427+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 171200 ms [2023-01-13T23:09:50.427+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:50.427+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:50.430+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:50.430+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:50.430+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:50.430+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 164, above threshold 10. [2023-01-13T23:09:51.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 172248 ms [2023-01-13T23:09:51.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:51.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:51.478+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:51.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:51.479+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:51.479+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 165, above threshold 10. [2023-01-13T23:09:52.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 173298 ms [2023-01-13T23:09:52.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:52.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:52.529+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:52.534+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:52.534+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:52.538+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 166, above threshold 10. [2023-01-13T23:09:53.575+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 174348 ms [2023-01-13T23:09:53.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:53.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:53.579+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:53.579+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:53.580+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:53.580+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 167, above threshold 10. [2023-01-13T23:09:54.626+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 175399 ms [2023-01-13T23:09:54.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:54.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:54.629+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:54.633+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:54.633+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:54.634+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 168, above threshold 10. [2023-01-13T23:09:55.675+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 176448 ms [2023-01-13T23:09:55.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:55.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:55.679+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:55.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:55.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:55.679+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 169, above threshold 10. [2023-01-13T23:09:56.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 177498 ms [2023-01-13T23:09:56.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:56.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:56.729+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:56.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:56.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:56.729+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 170, above threshold 10. [2023-01-13T23:09:57.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 178548 ms [2023-01-13T23:09:57.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:57.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:57.778+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:57.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:57.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:57.779+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 171, above threshold 10. [2023-01-13T23:09:58.825+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 179598 ms [2023-01-13T23:09:58.826+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:58.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:58.829+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:58.830+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:58.830+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:58.830+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 172, above threshold 10. [2023-01-13T23:09:59.355+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:09:59.356+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:09:59.356+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:09:59.359+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:59.359+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:59.359+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:09:59.875+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 180648 ms [2023-01-13T23:09:59.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:09:59.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:09:59.878+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:09:59.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:09:59.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:09:59.879+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 173, above threshold 10. [2023-01-13T23:10:00.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 181698 ms [2023-01-13T23:10:00.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:00.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:00.929+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:00.930+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:00.930+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:00.930+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 174, above threshold 10. [2023-01-13T23:10:01.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 182748 ms [2023-01-13T23:10:01.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:01.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:01.980+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:01.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:01.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:01.980+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 175, above threshold 10. [2023-01-13T23:10:03.025+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 183798 ms [2023-01-13T23:10:03.026+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:03.027+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:03.037+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:03.038+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:03.038+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:03.038+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 176, above threshold 10. [2023-01-13T23:10:04.075+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 184848 ms [2023-01-13T23:10:04.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:04.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:04.078+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:04.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:04.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:04.079+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 177, above threshold 10. [2023-01-13T23:10:05.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 185898 ms [2023-01-13T23:10:05.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:05.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:05.129+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:05.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:05.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:05.129+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 178, above threshold 10. [2023-01-13T23:10:06.181+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 186950 ms [2023-01-13T23:10:06.181+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:06.182+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:06.184+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:06.184+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:06.184+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:06.184+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 179, above threshold 10. [2023-01-13T23:10:07.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 187998 ms [2023-01-13T23:10:07.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:07.226+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:07.228+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:07.229+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:07.229+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:07.229+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 180, above threshold 10. [2023-01-13T23:10:08.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 189048 ms [2023-01-13T23:10:08.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:08.276+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:08.278+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:08.278+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:08.278+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:08.278+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 181, above threshold 10. [2023-01-13T23:10:09.326+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 190098 ms [2023-01-13T23:10:09.326+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:09.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:09.329+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:09.330+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:09.330+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:09.330+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 182, above threshold 10. [2023-01-13T23:10:10.377+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 191150 ms [2023-01-13T23:10:10.377+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:10.377+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:10.380+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:10.381+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:10.381+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:10.381+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 183, above threshold 10. [2023-01-13T23:10:11.425+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 192198 ms [2023-01-13T23:10:11.426+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:11.426+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:11.428+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:11.428+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:11.434+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:11.434+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 184, above threshold 10. [2023-01-13T23:10:12.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 193248 ms [2023-01-13T23:10:12.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:12.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:12.479+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:12.480+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:12.480+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:12.480+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 185, above threshold 10. [2023-01-13T23:10:13.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 194298 ms [2023-01-13T23:10:13.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:13.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:13.540+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:13.540+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:13.540+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:13.540+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 186, above threshold 10. [2023-01-13T23:10:14.364+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:10:14.365+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:10:14.365+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:10:14.370+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:14.370+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:14.371+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:10:14.576+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 195349 ms [2023-01-13T23:10:14.577+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:14.577+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:14.579+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:14.580+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:14.580+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:14.580+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 187, above threshold 10. [2023-01-13T23:10:15.626+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 196398 ms [2023-01-13T23:10:15.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:15.626+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:15.629+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:15.630+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:15.630+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:15.630+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 188, above threshold 10. [2023-01-13T23:10:16.675+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 197448 ms [2023-01-13T23:10:16.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:16.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:16.678+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:16.678+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:16.678+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:16.678+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 189, above threshold 10. [2023-01-13T23:10:17.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 198498 ms [2023-01-13T23:10:17.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:17.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:17.729+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:17.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:17.729+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:17.729+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 190, above threshold 10. [2023-01-13T23:10:18.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 199548 ms [2023-01-13T23:10:18.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:18.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:18.779+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:18.780+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:18.780+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:18.780+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 191, above threshold 10. [2023-01-13T23:10:19.825+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 200598 ms [2023-01-13T23:10:19.826+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:19.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:19.829+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:19.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:19.829+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:19.829+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 192, above threshold 10. [2023-01-13T23:10:20.875+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 201648 ms [2023-01-13T23:10:20.876+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:20.876+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:20.879+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:20.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:20.879+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:20.879+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 193, above threshold 10. [2023-01-13T23:10:21.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 202698 ms [2023-01-13T23:10:21.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:21.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:21.928+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:21.928+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:21.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:21.929+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 194, above threshold 10. [2023-01-13T23:10:22.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 203748 ms [2023-01-13T23:10:22.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:22.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:22.979+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:22.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:22.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:22.980+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 195, above threshold 10. [2023-01-13T23:10:24.025+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 204798 ms [2023-01-13T23:10:24.026+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:24.026+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:24.034+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:24.034+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:24.034+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:24.034+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 196, above threshold 10. [2023-01-13T23:10:25.075+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 205848 ms [2023-01-13T23:10:25.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:25.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:25.079+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:25.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:25.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:25.080+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 197, above threshold 10. [2023-01-13T23:10:26.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 206898 ms [2023-01-13T23:10:26.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:26.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:26.129+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:26.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:26.129+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:26.129+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 198, above threshold 10. [2023-01-13T23:10:27.175+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 207948 ms [2023-01-13T23:10:27.176+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:27.176+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:27.178+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:27.179+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:27.179+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:27.179+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 199, above threshold 10. [2023-01-13T23:10:28.225+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 208998 ms [2023-01-13T23:10:28.226+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:28.226+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:28.228+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:28.229+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:28.229+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:28.229+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 200, above threshold 10. [2023-01-13T23:10:29.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 210048 ms [2023-01-13T23:10:29.276+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:29.276+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:29.280+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:29.280+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:29.280+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:29.281+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 201, above threshold 10. [2023-01-13T23:10:29.371+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:10:29.372+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:10:29.372+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:10:29.375+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:29.375+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:29.375+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:10:30.325+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 211098 ms [2023-01-13T23:10:30.325+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:30.326+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:30.328+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:30.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:30.329+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:30.329+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 202, above threshold 10. [2023-01-13T23:10:31.375+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 212148 ms [2023-01-13T23:10:31.376+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:31.376+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:31.378+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:31.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:31.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:31.379+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 203, above threshold 10. [2023-01-13T23:10:32.429+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 213202 ms [2023-01-13T23:10:32.429+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:32.430+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:32.443+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:32.444+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:32.444+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:32.444+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 204, above threshold 10. [2023-01-13T23:10:33.475+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 214248 ms [2023-01-13T23:10:33.476+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:33.476+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:33.479+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:33.480+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:33.480+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:33.480+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 205, above threshold 10. [2023-01-13T23:10:34.525+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 215298 ms [2023-01-13T23:10:34.526+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:34.526+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:34.528+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:34.528+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:34.528+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:34.528+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 206, above threshold 10. [2023-01-13T23:10:35.575+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 216348 ms [2023-01-13T23:10:35.576+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:35.576+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:35.578+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:35.578+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:35.578+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:35.578+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 207, above threshold 10. [2023-01-13T23:10:36.625+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 217398 ms [2023-01-13T23:10:36.626+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:36.627+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:36.633+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:36.633+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:36.634+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:36.634+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 208, above threshold 10. [2023-01-13T23:10:37.675+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 218448 ms [2023-01-13T23:10:37.676+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:37.676+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:37.679+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:37.679+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:37.680+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:37.681+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 209, above threshold 10. [2023-01-13T23:10:38.725+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 219498 ms [2023-01-13T23:10:38.726+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:38.726+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:38.728+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:38.728+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:38.728+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:38.729+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 210, above threshold 10. [2023-01-13T23:10:39.775+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 220548 ms [2023-01-13T23:10:39.776+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:39.776+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:39.778+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:39.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:39.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:39.779+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 211, above threshold 10. [2023-01-13T23:10:40.781+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 221554 ms [2023-01-13T23:10:40.781+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:40.781+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:40.789+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:40.789+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:40.789+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:40.789+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 212, above threshold 10. [2023-01-13T23:10:41.825+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 222598 ms [2023-01-13T23:10:41.826+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:41.826+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:41.829+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:41.830+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:41.830+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:41.830+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 213, above threshold 10. [2023-01-13T23:10:42.876+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 223649 ms [2023-01-13T23:10:42.877+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:42.877+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:42.880+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:42.881+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:42.881+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:42.882+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 214, above threshold 10. [2023-01-13T23:10:43.925+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 224698 ms [2023-01-13T23:10:43.926+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:43.926+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:43.929+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:43.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:43.929+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:43.930+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 215, above threshold 10. [2023-01-13T23:10:44.376+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=77a0f2ff-0be2-40f5-8212-a8fa72ce54b4, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2023-01-13T23:10:44.376+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 [2023-01-13T23:10:44.376+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/77a0f2ff-0be2-40f5-8212-a8fa72ce54b4/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2023-01-13T23:10:44.379+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:44.380+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:44.380+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-01-13T23:10:44.975+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 225748 ms [2023-01-13T23:10:44.976+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:44.976+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:44.979+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:44.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:44.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:44.980+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 216, above threshold 10. [2023-01-13T23:10:46.029+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 226802 ms [2023-01-13T23:10:46.029+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:46.029+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:46.031+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:46.032+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:46.032+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:46.032+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 217, above threshold 10. [2023-01-13T23:10:47.075+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 227848 ms [2023-01-13T23:10:47.076+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:47.076+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:47.078+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:47.078+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:47.079+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:47.079+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 218, above threshold 10. [2023-01-13T23:10:48.125+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 228898 ms [2023-01-13T23:10:48.126+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:48.126+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:48.128+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:48.128+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:48.128+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:48.128+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 219, above threshold 10. [2023-01-13T23:10:49.175+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 229948 ms [2023-01-13T23:10:49.176+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-01-13T23:10:49.176+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2023-01-13T23:10:49.178+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2023-01-13T23:10:49.178+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-13T23:10:49.178+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2023-01-13T23:10:49.178+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 220, above threshold 10. [2023-